Re: [PATCH 1/3] MAINTAINERS: Introduce V: field for required tests

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Theodore,

On 11/20/23 22:51, Theodore Ts'o wrote:
On Mon, Nov 20, 2023 at 02:30:49PM +0100, Ricardo Cañuelo wrote:

This is not trivial because tests vary a lot and we'd first need to
define which artifacts to link to, and because whatever is linked (test
commands, output log, results summary) would need to be stored
forever. But since we're doing that already for basically all kernel
mailing lists, I wonder if something like "public-inbox for test
results" could be possible some day.

What we have at work is a way to upload the test results summary
(e.g., just KTAP result lines, or the xfstests junit XML) along with
test run metadata (e.g., what was the kernel commit on which the test
was run, and the test hardware), and this would be stored permanently.
Test artifacts is also preserved but for a limited amount of time
(e.g., some number of months or a year).

The difference in storage lifetimes is because the junit XML file
might be a few kilobytes to tens of kilobytes. but the test artifacts
might be a few megabytes to tens of megabytes.

Of course once you have this data, it becomes possible to detect when
a test may have regressed, or to detect flaky tests, and perhaps to
figure out if certain hardware configurations or kernel configurations
are more likely to trigger a particular test to fail.  So having all
of this data stored centrally would be really cool.  The only question
is who might be able to create such an infrastructure, and be able to
pay for the ongoing development and operational costs....

Yes, I agree, having public result storage would be awesome. I think KCIDB is relatively-well positioned to take on some of that work. We have the POC dashboard already. Well, *had* until somebody scraped it and exhausted our cloud budget, I'm working on making it cheaper before bringing it back.

Meanwhile you can get a glimpse of it in one of my slide decks:
https://static.sched.com/hosted_files/eoss2023/ef/Kernel%20CI%20%E2%80%93%20How%20Far%20Can%20It%20Go%20%E2%80%93%20EOSS%202023.pdf

We also have an artifact-mirroring system (quite basic at the moment), expecting the submitter to already have the files hosted somewhere. We would need to add a file upload service to that.

Finally, I'm considering opening up submissions to (Google-authenticated) public, as opposed to just CI systems, so we could support this. That's still more work though.

Regarding analysis, I have plenty of ideas for that too, and even more ideas are explored by others lately. I just need to bring back the dashboard and find the time for all of it :D

Anyone interested in helping with any of this?

Nick





[Index of Archives]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Share Photos]     [IDE]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux ATA RAID]     [Samba]     [Device Mapper]

  Powered by Linux