On Mon, 19 Sep 2022 17:25:38 -0700 Daniel Latypov <dlatypov@xxxxxxxxxx> wrote: > On Mon, Sep 19, 2022 at 1:43 PM Isabella Basso <isabbasso@xxxxxxxxxx> wrote: > > >> + * "(K)TAP version XX" should be the first line on all (sub)tests as per > > >> + * https://www.kernel.org/doc/html/latest/dev-tools/ktap.html#version-lines > > >> + * but actually isn't, as it currently depends on whoever writes the > > >> + * test to print this info > > > > > > FYI: we're really trying to fix cases of "missing version lines", > > > largely by making the kunit_test_suites() macro work in more > > > circumstances. > > > > > > So while it may be worth still handling the case where this is > > > missing, I don't think there are any tests in the latest versions of > > > the kernel which should have this missing. > > > > I’m not sure if I totally get how these work. Every time I run a KUnit test I > > get something like this: https://pastebin.com/7Ff31PMC > > > > As you can see it has been loaded as a module, just like we intend to do it > > from IGT, and I see no version lines whatsoever. Am I doing something wrong? > > You're doing everything right. > > The problem is we only print the version line for the *built-in* tests. > It never gets printed for tests in loadable modules. IGT tests require loadable modules, as some tests check if bind/unbind and load/unload works. Also, some tests require modprobing drivers with different parameters. Unfortunately, kunit.py is currently broken on such cases: its parser simply ignores everything if the subtest doesn't have a TAP version just before it. See: $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg [12:37:05] ============================================================ [12:37:05] [ERROR] Test: main: 0 tests run! [12:37:05] ============================================================ [12:37:05] Testing complete. Ran 0 tests: errors: 1 Here, I loaded the test module twice. I can force it to parse it by doing $ (dmesg|grep "TAP version"; dmesg|grep -A9999 intel_i915_mock) >logs But then, it gets confused with two subtests with the same name: $ ./tools/testing/kunit/kunit.py parse logs [12:37:34] ============================================================ [12:37:34] ============== intel_i915_mock (18 subtests) =============== [12:37:34] [PASSED] sanitycheck [12:37:34] [PASSED] shmem [12:37:34] [PASSED] fence [12:37:34] [PASSED] scatterlist [12:37:34] [PASSED] syncmap [12:37:34] [PASSED] uncore [12:37:34] [PASSED] ring [12:37:34] [PASSED] engine [12:37:34] [PASSED] timelines [12:37:34] [PASSED] requests [12:37:34] [PASSED] objects [12:37:34] [PASSED] phys [12:37:34] [PASSED] dmabuf [12:37:34] [PASSED] vma [12:37:34] [PASSED] evict [12:37:34] [PASSED] gtt [12:37:34] [PASSED] hugepages [12:37:34] [PASSED] memory_region [12:37:34] ================= [PASSED] intel_i915_mock ================= [12:37:34] ============== intel_i915_mock (18 subtests) =============== [12:37:34] [PASSED] sanitycheck [12:37:34] [PASSED] shmem [12:37:34] [PASSED] fence [12:37:34] [PASSED] scatterlist [12:37:34] [PASSED] syncmap [12:37:34] [PASSED] uncore [12:37:34] [PASSED] ring [12:37:34] [PASSED] engine [12:37:34] [PASSED] timelines [12:37:34] [PASSED] requests [12:37:34] [PASSED] objects [12:37:34] [PASSED] phys [12:37:34] [PASSED] dmabuf [12:37:34] [PASSED] vma [12:37:34] [PASSED] evict [12:37:34] [PASSED] gtt [12:37:34] [PASSED] hugepages [12:37:34] [PASSED] memory_region [12:37:34] [ERROR] Test: intel_i915_mock: Expected test number 2 but found 1 [12:37:34] ================= [PASSED] intel_i915_mock ================= [12:37:34] ============================================================ [12:37:34] Testing complete. Ran 36 tests: passed: 36, errors: 1 > Some more details below, if interested. > Specifically, it happens in > https://elixir.bootlin.com/linux/latest/C/ident/kunit_print_tap_header > > What David is referring to is we had tests that weren't in modules, > but didn't use the normal built-in code path. Those were also missing > TAP version lines. > But KUnit needs to do better here in the case of modules. > > The goal was that you'd be able to parse the result of module based > tests by something like > $ cat /sys/kernel/debug/kunit/*/results | ./tools/testing/kunit/kunit.py parse > but this doesn't work because of the lack of a version line. At least here, debug/kunit doesn't exist: sudo ls -lctra /sys/kernel/debug/|grep kunit > > If we add it to each module's test results, then we'll need to update > the parse to allow multiple verison lines, I think. Yeah, the parser is currently broken when used with modules. Regards, Mauro
