Hi Theodore,
On 11/20/23 22:51, Theodore Ts'o wrote:
On Mon, Nov 20, 2023 at 02:30:49PM +0100, Ricardo Cañuelo wrote:
This is not trivial because tests vary a lot and we'd first need to
define which artifacts to link to, and because whatever is linked (test
commands, output log, results summary) would need to be stored
forever. But since we're doing that already for basically all kernel
mailing lists, I wonder if something like "public-inbox for test
results" could be possible some day.
What we have at work is a way to upload the test results summary
(e.g., just KTAP result lines, or the xfstests junit XML) along with
test run metadata (e.g., what was the kernel commit on which the test
was run, and the test hardware), and this would be stored permanently.
Test artifacts is also preserved but for a limited amount of time
(e.g., some number of months or a year).
The difference in storage lifetimes is because the junit XML file
might be a few kilobytes to tens of kilobytes. but the test artifacts
might be a few megabytes to tens of megabytes.
Of course once you have this data, it becomes possible to detect when
a test may have regressed, or to detect flaky tests, and perhaps to
figure out if certain hardware configurations or kernel configurations
are more likely to trigger a particular test to fail. So having all
of this data stored centrally would be really cool. The only question
is who might be able to create such an infrastructure, and be able to
pay for the ongoing development and operational costs....
Yes, I agree, having public result storage would be awesome. I think KCIDB is
relatively-well positioned to take on some of that work. We have the POC
dashboard already. Well, *had* until somebody scraped it and exhausted our
cloud budget, I'm working on making it cheaper before bringing it back.
Meanwhile you can get a glimpse of it in one of my slide decks:
https://static.sched.com/hosted_files/eoss2023/ef/Kernel%20CI%20%E2%80%93%20How%20Far%20Can%20It%20Go%20%E2%80%93%20EOSS%202023.pdf
We also have an artifact-mirroring system (quite basic at the moment),
expecting the submitter to already have the files hosted somewhere. We would
need to add a file upload service to that.
Finally, I'm considering opening up submissions to (Google-authenticated)
public, as opposed to just CI systems, so we could support this. That's still
more work though.
Regarding analysis, I have plenty of ideas for that too, and even more ideas
are explored by others lately. I just need to bring back the dashboard and
find the time for all of it :D
Anyone interested in helping with any of this?
Nick
