On Mon, Mar 28, 2022 at 2:57 AM Maxime Ripard <maxime@xxxxxxxxxx> wrote:
>
> Hi,
>
> On Fri, Mar 25, 2022 at 05:36:25PM -0500, Daniel Latypov wrote:
> > On Mon, Feb 28, 2022 at 4:47 AM Maxime Ripard <maxime@xxxxxxxxxx> wrote:
> > >
> > > On Fri, Feb 25, 2022 at 01:29:03PM -0800, Daniel Latypov wrote:
> > > > On Fri, Feb 25, 2022 at 5:23 AM Maxime Ripard <maxime@xxxxxxxxxx> wrote:
> > > > >
> > > > > Hi Daniel,
> > > > >
> > > > > On Wed, Feb 23, 2022 at 02:50:59PM -0800, Daniel Latypov wrote:
> > > > > > On Wed, Feb 23, 2022 at 2:56 AM Maxime Ripard <maxime@xxxxxxxxxx> wrote:
> > > > > > >
> > > > > > > Let's test various parts of the rate-related clock API with the kunit
> > > > > > > testing framework.
> > > > > > >
> > > > > > > Cc: kunit-dev@xxxxxxxxxxxxxxxx
> > > > > > > Suggested-by: Stephen Boyd <sboyd@xxxxxxxxxx>
> > > > > > > Signed-off-by: Maxime Ripard <maxime@xxxxxxxxxx>
> > > > > >
> > > > > > Tested-by: Daniel Latypov <dlatypov@xxxxxxxxxx>
> > > > > >
> > > > > > Looks good to me on the KUnit side.
> > > > > > Two small nits below.
> > > > > >
> > > > > > FYI, I computed the incremental coverage for this series, i.e.:
> > > > > > 1) applied the full series
> > > > > > 2) computed the absolute coverage
> > > > > >
> > > > > > $ ./tools/testing/kunit/kunit.py run --kunitconfig=drivers/clk
> > > > > > --make_options=CC=/usr/bin/gcc-6 --kconfig_add=CONFIG_DEBUG_KERNEL=y
> > > > > > --kconfig_add=CONFIG_DEBUG_INFO=y --kconfig_add=CONFIG_GCOV=y
> > > > >
> > > > > I built a docker container based on ubuntu 18.04 to have gcc6 and
> > > > > python3.7, but this doesn't seem to be working, I'm not entirely sure why:
> > > > >
> > > > > [13:11:22] Configuring KUnit Kernel ...
> > > > > Regenerating .config ...
> > > > > Populating config with:
> > > > > $ make ARCH=um olddefconfig CC=/usr/bin/gcc-6 O=.kunit
> > > > > ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config.
> > > > > This is probably due to unsatisfied dependencies.
> > > > > Missing: CONFIG_DEBUG_INFO=y, CONFIG_GCOV=y
> > > > > Note: many Kconfig options aren't available on UML. You can try running on a different architecture with something like "--arch=x86_64".
> > > >
> > > > Did you perhaps drop CONFIG_DEBUG_KERNEL=y?
> > > > Need to add 3 config options in total for coverage.
> > > >
> > > > If I tweak the command I ran above but drop CONFIG_DEBUG_KERNEL=y, I
> > > > get the error message you get:
> > > >
> > > > $ ./tools/testing/kunit/kunit.py run --kunitconfig=drivers/clk
> > > > --make_options=CC=/usr/bin/gcc-6 --kconfig_add=CONFIG_DEBUG_INFO=y
> > > > --kconfig_add=CONFIG_GCOV=y
> > > > ...
> > > > Missing: CONFIG_DEBUG_INFO=y, CONFIG_GCOV=y
> > > > Note: many Kconfig options aren't available on UML. You can try
> > > > running on a different architecture with something like
> > > > "--arch=x86_64".
> > >
> > > It looks to me that it's more that DEBUG_INFO isn't enabled.
> >
> > Sorry for the very delayed response.
> > I was largely getting internet over mobile data around when this email
> > came in and didn't want to try and download docker images over that.
> >
> > It looks like that there was another change that is now merged into
> > Linus' tree that causes this.
> >
> > I found that adding this helped (thanks David Gow)
> > --kconfig_add=DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
> >
> > Running against --kunitconfig=lib/kunit, my final coverage result is
> >
> > Overall coverage rate:
> > lines......: 13.6% (18004 of 132055 lines)
> > functions..: 15.7% (1885 of 12010 functions)
> >
> > Can you give that a shot and see if it works?
>
> It does fix the configuration issue, but I'm not able to run the tests either:
>
> [07:53:51] Configuring KUnit Kernel ...
> Generating .config ...
> Populating config with:
> $ make ARCH=um olddefconfig O=/home/max/out
> [07:53:53] Building KUnit Kernel ...
> Populating config with:
> $ make ARCH=um olddefconfig O=/home/max/out
> Building with:
> $ make ARCH=um --jobs=16 O=/home/max/out
> [07:54:09] Starting KUnit Kernel (1/1)...
> [07:54:09] ============================================================
> [07:54:09] [ERROR] Test : invalid KTAP input!
> [07:54:09] ============================================================
> [07:54:09] Testing complete. Passed: 0, Failed: 0, Crashed: 0, Skipped: 0, Errors: 1
> [07:54:09] Elapsed time: 18.486s total, 2.430s configuring, 16.052s building, 0.003s running
>
>
> I've tried to remove all the coverage from the equation, and I get the
> same issue if I only run kunit run from inside the container, but it
> works fine outside. So I guess it's my setup that is broken. Is there
> some way to debug what could be going wrong there?
kunit.py is failing to find any test results from the raw kernel dmesg output.
That is stored in $BUILD_DIR/test.log, so /home/max/out/test.log.
(You can also have kunit.py print this out instead with `kunit.py run
--raw_output`)
It looks like it's specifically not finding the (K)TAP header.
Here's a snippet of what you'd expect to see:
<bunch of boot stuff>
printk: console [mc-1] enabled
TAP version 14
1..9
# Subtest: clk-test
1..4
ok 1 - clk_test_get_rate
ok 2 - clk_test_set_get_rate
<more kunit output in similar format>
Here's the failing lines in kunit_parser.py
805 def parse_run_tests(kernel_output: Iterable[str]) -> Test:
817 lines = extract_tap_lines(kernel_output) <= lines
after "K?TAP version"
818 test = Test()
819 if not lines:
820 test.add_error('invalid KTAP input!') <= this error msg
821 test.status = TestStatus.FAILURE_TO_PARSE_TESTS
Normally the issue people run into is that KUnit prints a header but
has no tests.
That's a different error than what you see here.
It seems like we're either not running this func
206 static void kunit_exec_run_tests(struct suite_set *suite_set)
207 {
208 struct kunit_suite * const * const *suites;
209
210 kunit_print_tap_header(suite_set); <= not hitting this line?
211
212 for (suites = suite_set->start; suites <
suite_set->end; suites++)
213 __kunit_test_suites_init(*suites);
214 }
or maybe the output got mangled somehow?
Daniel
