On Sun, Oct 06, 2024 at 12:33:51AM -0400, Kent Overstreet wrote:
>
> Correct me if I'm wrong, but your system isn't available to the
> community, and I haven't seen a CI or dashboard for kdevops?
It's up on github for anyone to download, and I've provided pre-built
test appliance so people don't have to have downloaded xfstests and
all of its dependencies and build it from scratch. (That's been
automated, of course, but the build infrastructure is setup to use a
Debian build chroot, and with the precompiled test appliances, you can
use my test runner on pretty much any Linux distribution; it will even
work on MacOS if you have qemu built from macports, although for now
you have to build the kernel on Linux distro using Parallels VM[1].)
I'll note that IMHO making testing resources available to the
community isn't really the bottleneck. Using cloud resources,
especially if you spin up the VM's only when you need to run the
tests, and shut them down once the test is complete, which
gce-xfstests does, is actually quite cheap. At retail prices, running
a dozen ext4 file system configurations against xfstests's "auto"
group will take about 24 hours of VM time, and including the cost of
the block devices, costs just under two dollars USD. Because the
tests are run in parallel, the total wall clock time to run all of the
tests is about two and a half hours. Running the "quick" group on a
single file system configuration costs pennies. So the $300 of free
GCE credits will actually get someone pretty far!
No, the bottleneck is having someone knowledgeable enough to interpret
the test results and then finding the root cause of the failures.
This is one of the reasons why I haven't stressed all that much about
dashboards. Dashboards are only useful if the right person(s) is
looking at them. That's why I've been much more interested in making
it stupidly easy to run tests on someone's local resources, e.g.:
https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md
In fact, for most people, the entry point that I envision as being
most interesting is that they download the kvm-xfstests, and following
the instructions in the quickstart, so they can run "kvm-xfstests
smoke" before sending me an ext4 patch. Running the smoke test only
takes 15 minutes using qemu, and it's much more convenient for them to
run that on their local machine than to trigger the test on some
remote machine, whether it's in the cloud or someone's remote test
server.
In any case, that's why I haven't been interesting in working with
your test infrastructure; I have my own, and in my opinion, my
approach is the better one to make available to the community, and so
when I have time to improve it, I'd much rather work on
{kvm,gce,android}-xfstests.
Cheers,
- Ted
[1] Figuring out how to coerce the MacOS toolchain to build the Linux
kernel would be cool if anyone ever figures it out. However, I *have*
done kernel development using a Macbook Air M2 while on a cruise ship
with limited internet access, building the kernel using a Parallels VM
running Debian testing, and then using qemu from MacPorts to avoid the
double virtualization performance penalty to run xfstests to test the
freshly-built arm64 kernel, using my xfstests runner -- and all of
this is available on github for anyone to use.
