On Thu, Jan 11, 2024 at 12:38:57PM -0500, Kent Overstreet wrote: > On Thu, Jan 11, 2024 at 03:35:40PM +0000, Mark Brown wrote: > > IME the actually running the tests bit isn't usually *so* much the > > issue, someone making a new test runner and/or output format does mean a > > bit of work integrating it into infrastructure but that's more usually > > annoying than a blocker. > No, the proliferation of test runners, test output formats, CI systems, > etc. really is an issue; it means we can't have one common driver that > anyone can run from the command line, and instead there's a bunch of > disparate systems with patchwork integration and all the feedback is nag > emails - after you've finished whan you were working on instead of > moving on to the next thing - with no way to get immediate feedback. It's certainly an issue and it's much better if people do manage to fit their tests into some existing thing but I'm not convinced that's the big reason why you have a bunch of different systems running separately and doing different things. For example the enterprise vendors will naturally tend to have a bunch of server systems in their labs and focus on their testing needs while I know the Intel audio CI setup has a bunch of laptops, laptop like dev boards and things in there with loopback audio cables and I think test equipment plugged in and focuses rather more on audio. My own lab is built around on systems I can be in the same room as without getting too annoyed and does things I find useful, plus using spare bandwidth for KernelCI because they can take donated lab time. I think there's a few different issues you're pointing at here: - Working out how to run relevant tests for whatever area of the kernel you're working on on whatever hardware you have to hand. - Working out exactly what other testers will do. - Promptness and consistency of feedback from other testers. - UI for getting results from other testers. and while it really sounds like your main annoyances are the bits with other test systems it really seems like the test runner bit is mainly for the first issue, possibly also helping with working out what other testers are going to do. These are all very real issues. > And it's because building something shiny and new is the fun part, no > one wants to do the grungy integration work. I think you may be overestimating people's enthusiasm for writing test stuff there! There is NIH stuff going on for sure but lot of the time when you look at something where people have gone off and done their own thing it's either much older than you initially thought and predates anything they might've integrated with or there's some reason why none of the existing systems fit well. Anecdotally it seems much more common to see people looking for things to reuse in order to save time than it is to see people going off and reinventing the world. > > > example tests, example output: > > > https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/single_device.ktest > > > https://evilpiepirate.org/~testdashboard/ci?branch=bcachefs-testing > > For example looking at the sample test there it looks like it needs > > among other things mkfs.btrfs, bcachefs, stress-ng, xfs_io, fio, mdadm, > > rsync > Getting all that set up by the end user is one command: > ktest/root_image create > and running a test is one morecommand: > build-test-kernel run ~/ktest/tests/bcachefs/single_device.ktest That does assume that you're building and running everything directly on the system under test and are happy to have the test in a VM which isn't an assumption that holds universally, and also that whoever's doing the testing doesn't want to do something like use their own distro or something - like I say none of it looks too unreasonable for filesystems. > > and a reasonably performant disk with 40G of space available. > > None of that is especially unreasonable for a filesystems test but it's > > all things that we need to get onto the system where we want to run the > > test and there's a lot of systems where the storage requirements would > > be unsustainable for one reason or another. It also appears to take > > about 33000s to run on whatever system you use which is distinctly > > non-trivial. > Getting sufficient coverage in filesystem land does take some amount of > resources, but it's not so bad - I'm leasing 80 core ARM64 machines from > Hetzner for $250/month and running 10 test VMs per machine, so it's > really not that expensive. Other subsystems would probably be fine with > less resources. Some will be, some will have more demanding requirements especially when you want to test on actual hardware rather than in a VM. For example with my own test setup which is more focused on hardware the operating costs aren't such a big deal but I've got boards that are for various reasons irreplaceable, often single instances of boards (which makes scheduling a thing) and for some of the tests I'd like to get around to setting up I need special physical setup. Some of the hardware I'd like to cover is only available in machines which are in various respects annoying to automate, I've got a couple of unused systems waiting for me to have sufficient bandwidth to work out how to automate them. Either way I don't think the costs are trival enough to be completely handwaved away. I'd also note that the 9 hour turnaround time for that test set you're pointing at isn't exactly what I'd associate with immediate feedback.
Attachment:
signature.asc
Description: PGP signature
