On Wed, Feb 2, 2022 at 1:38 PM Frank Rowand <frowand.list@xxxxxxxxx> wrote: > > On 2/2/22 5:31 AM, Naresh Kamboju wrote: > > Linaro started doing Linux kernel Functional Validation (LKFT). > > As part of LKFT recently we have enabled CONFIG_OF_UNITTEST=y in our > > daily test CI. > > > > The output of the test looks as below. The current problem is that we > > have a hard time to see (grep) pass/fail for each individual test. We > > only see a summary at the end with x pass and y fails. > > The FAIL messages are printed at loglevel KERN_ERR. The pass messages > are printed at loglevel KERN_DEBUG. To see the pass messages, set the > loglevel to allow debug output. > > Unfortunately this can add lots of debug output, unless you use dynamic > debug to only enable debug for drivers/of/unittest.o. There are only > a few other pr_debug() messages in unittest. > > I think a better solution would be to add a config option, something > like CONFIG_OF_UNITTEST_VERBOSE, that would print the pass messages > at loglevel KERN_ERR. I'll submit a patch for that and see what the > review responses are. > > > We would like to get your opinion of how hard it would be to include > > that in the output per test. Maybe like TAP version 14? > > Another question would be how hard do you think it would be to rewrite > > this to a kunit test, if even applicable? I have provided the kunit > > output links at the end of this email. > > Devicetree unittests were suggested as a good candidate as a first > test to convert to kunit when kunit was implemented. Brendan tried > to convert it, and we quickly saw that it was not a good candidate. > Devicetree unittests do not fit the unit test mold; they are a very > different creature. Brendan has a good term for this type of test > (Brendan, was it "acceptance" test?). I understood that it was either an integration test or end-to-end test (probably an integration test): https://lkml.org/lkml/2019/3/21/1124 Standardizing integration tests in the kernel is still something that hasn't happened yet, but there are some examples of integration tests being written in KUnit (the KASAN KUnit test is probably the most notable example). There are definitely some others written in kselftest. It's kind of a tough area because integration tests are kind of defined by being in between unit tests and end-to-end tests. > > Test output: > > ------------ > > [ 0.000000] Booting Linux on physical CPU 0x0000000100 [0x410fd033] > > [ 0.000000] Linux version 5.17.0-rc1-next-20220127 > > (tuxmake@tuxmake) (aarch64-linux-gnu-gcc (Debian 11.2.0-9) 11.2.0, GNU > > ld (GNU Binutils for Debian) 2.37) #1 SMP PREEMPT @1643255563 > > [ 0.000000] Machine model: ARM Juno development board (r2) > > > > <trimmed output> > > > > [ 3.285226] ### dt-test ### start of unittest - you will see error messages > > [ 3.293269] ### dt-test ### EXPECT \ : Duplicate name in > > testcase-data, renamed to \"duplicate-name#1\" > > [ 3.293456] Duplicate name in testcase-data, renamed to \"duplicate-name#1\" > > [ 3.313367] ### dt-test ### EXPECT / : Duplicate name in > > testcase-data, renamed to \"duplicate-name#1\" > > [ 3.314709] ### dt-test ### EXPECT \ : OF: > > /testcase-data/phandle-tests/consumer-a: could not get > > #phandle-cells-missing for /testcase-data/phandle-tests/provider1 > > [ 3.323968] OF: /testcase-data/phandle-tests/consumer-a: could not > > get #phandle-cells-missing for /testcase-data/phandle-tests/provider1 > > > > <trimmed output> > > > > [ 5.118400] ### dt-test ### EXPECT / : OF: overlay: ERROR: multiple > > fragments add and/or delete node > > /testcase-data-2/substation@100/motor-1/electric > > [ 5.121358] atkbd serio1: keyboard reset failed on 1c070000.kmi > > [ 5.134160] ### dt-test ### end of unittest - 257 passed, 0 failed > > > > > > Ref: > > Full test output of of-unittest > > https://lkft.validation.linaro.org/scheduler/job/4458582#L1019 > > https://lkft.validation.linaro.org/scheduler/job/4404330#L428 > > > > Kunit example test output that we are running in our daily CI loop. > > https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.10.y/build/v5.10.70/testrun/5965109/suite/kunit/tests/ > > > > Kunit Full test logs: > > https://lkft.validation.linaro.org/scheduler/job/3643324 > > > > https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.10.y/build/v5.10.70/testrun/5965109/suite/kunit/test/kunit_log_test/log > > > > > > -- > > Linaro LKFT > > https://lkft.linaro.org > > >
