On Tue, 11 Feb 2025 at 04:20, Yury Norov <yury.norov@xxxxxxxxx> wrote: > > On Mon, Feb 10, 2025 at 11:35:48AM -0800, John Hubbard wrote: > > On 2/9/25 11:54 PM, Geert Uytterhoeven wrote: > > > On Sat, 8 Feb 2025 at 18:53, Yury Norov <yury.norov@xxxxxxxxx> wrote: > > > > On Fri, Feb 07, 2025 at 03:14:01PM -0500, Tamir Duberstein wrote: > > > > > On 7/27/24 12:35 AM, Shuah Khan wrote: > > ... > > > > > The crux of the argument seems to be that the config help text is taken > > > > > to describe the author's intent with the fragment "at boot". I think > > > > > > IMO, "at boot" is a misnomer, as most tests can be either builtin > > > or modular. > > > > Right. > > > > > > > > > KUNIT is disabled in defconfig, at least on x86_64. It is also disabled > > > > on my Ubuntu 24.04 machine. If I take your patches, I'll be unable to > > > > OK so I just bought a shiny new test machine, and installed one of the > > big name distros on it, hoping they've moved ahead and bought into the kunit > > story... > > > > $ grep KUNIT /boot/config-6.8.0-52-generic > > # CONFIG_KUNIT is not set > > > > ...gagghh! No such luck. One more data point, in support of Yuri's complaint. :) > > > > > > > > I think distros should start setting CONFIG_KUNIT=m. > > > > Yes they should! kunit really does have important advantages for many use > > cases, including bitmaps here, and "CONFIG_KUNIT is not set" is the main > > obstacle. > > Hi John, Geert, Tamir, > > Can you please explain in details which advantages KUNIT brings to > the test_bitmap.c, find_bit_benchmark.c and other low-level tests? > First, I'd agree with Tamir and others who've pointed out that KUnit is not really intended for benchmarks, so I definitely don't want to oversell it for find_bit_benchmark.c and others (though it has been used for benchmarks in a few cases, it's not perfect). The main advantages of KUnit are that: - it's already used for a bunch of lib/ tests (so quite familiar to a number of contributors/users), - tests can be run at boot without needing any userspace at all (if built-in), - KUnit tests are already being automatically run as regression tests, and it's relatively easy to run "all tests" associated with a config - there's a whole bunch of tooling which automates configuring, building, and running a kernel with these tests, across several architectures (e.g., passing --arch arm64 or --arch m68k to automatically build and run the tests in QEMU for those architectures) - there's tooling to nicely print, format, and summarise the results of any number of suites. - and, as a result of the above, it's very quick and easy to run a test (or whole bunch of tests) in an isolated environment. KUnit also has very few runtime dependencies from a developer point-of-view (python, optionally QEMU). It is, however, less focused on the "debug a running system" case, hence the requirement for CONFIG_KUNIT to be enabled, and the tooling largely being focused on the "run all of the tests in UML or a VM" case. > I'm not strongly against moving under KUNIT's hat, but I do: > - respect commitment of my contributors, so I don't want to wipe git > history for no serious reason; It should be possible to preserve the git history during a port, though it admittedly will have a bunch of porting changes which make it more difficult to navigate with git blame. But it should at least be better than a total wipe. > - respect time of my testers, so no extra dependencies; I think KUnit is a bit of a mixed bag here. In many cases, KUnit tests can be run pretty quickly, either because the tester's environment already has CONFIG_KUNIT enabled, or possibly under some kind of virtualization with the tooling (either QEMU, or on x86, User-Mode-Linux). But, as mentioned above, I'd not consider it ideal for performance testing. I am very sympathetic to not wanting to add a dependency here. I think KUnit is a pretty minimal dependency as far as they go (it selects CONFIG_GLOB and nothing else, needs no userspace, and even the tooling only really requires python on top of the basic toolchain), but it's definitely a further dependency for the 'test-the-currently-running-kernel' use-case. > - respect time of reviewers. This is an area where I think KUnit could be an advantage, as it'd be easy for reviewers to simply run ./tools/testing/kunit/kunit.py run bitmap as a spot check on the code. And if they want to test on other architectures, the --arch option can be used to do so quickly. (And, of course, it can still be loaded as a module (or even built-in) on a physical system or other VM, albeit with the requirement for CONFIG_KUNIT.) As for distros enabling CONFIG_KUNIT by default, as mentioned, my understanding is that the Fedora/Red Hat distros are doing it, as are some branches of Android, but I doubt anyone else is. This makes sense: enabling CONFIG_KUNIT isn't free (though the overhead should be very low if no test is running, just a few static branches and an extra check in the module loader), and there's always been some reticence (which I've shared) in enabling test features on production kernels. So it's definitely something I was recommending against a few years ago. However, we've definitely made sure that CONFIG_KUNIT is less invasive, with things like hooks being kept behind static branches, and made the decision to have _running_ a test taint the kernel, so it should be relatively safe to just have the framework enabled (and even safe to have it loaded), and pretty obvious when a test is run. We've also got a bunch of extra options to disable running tests automatically when they're built-in/loaded, both at runtime and compile time. So I'd definitely be happy with other distros choosing to enable CONFIG_KUNIT by default now: I think it's much safer than it was, and you won't be going against the grain. But equally, I'd understand having it disabled in production: the cost, while very small, is nonzero. Cheers, -- David
