On 2/2/22 2:54 PM, Brendan Higgins wrote: > On Wed, Feb 2, 2022 at 1:38 PM Frank Rowand <frowand.list@xxxxxxxxx> wrote: >> >> On 2/2/22 5:31 AM, Naresh Kamboju wrote: >>> Linaro started doing Linux kernel Functional Validation (LKFT). >>> As part of LKFT recently we have enabled CONFIG_OF_UNITTEST=y in our >>> daily test CI. >>> >>> The output of the test looks as below. The current problem is that we >>> have a hard time to see (grep) pass/fail for each individual test. We >>> only see a summary at the end with x pass and y fails. >> >> The FAIL messages are printed at loglevel KERN_ERR. The pass messages >> are printed at loglevel KERN_DEBUG. To see the pass messages, set the >> loglevel to allow debug output. >> >> Unfortunately this can add lots of debug output, unless you use dynamic >> debug to only enable debug for drivers/of/unittest.o. There are only >> a few other pr_debug() messages in unittest. >> >> I think a better solution would be to add a config option, something >> like CONFIG_OF_UNITTEST_VERBOSE, that would print the pass messages >> at loglevel KERN_ERR. I'll submit a patch for that and see what the >> review responses are. >> >>> We would like to get your opinion of how hard it would be to include >>> that in the output per test. Maybe like TAP version 14? >>> Another question would be how hard do you think it would be to rewrite >>> this to a kunit test, if even applicable? I have provided the kunit >>> output links at the end of this email. >> >> Devicetree unittests were suggested as a good candidate as a first >> test to convert to kunit when kunit was implemented. Brendan tried >> to convert it, and we quickly saw that it was not a good candidate. >> Devicetree unittests do not fit the unit test mold; they are a very >> different creature. Brendan has a good term for this type of test >> (Brendan, was it "acceptance" test?). > > I understood that it was either an integration test or end-to-end test > (probably an integration test): https://lkml.org/lkml/2019/3/21/1124 Yes, thanks. Those are the terms I was trying to remember. -Frank > > Standardizing integration tests in the kernel is still something that > hasn't happened yet, but there are some examples of integration tests > being written in KUnit (the KASAN KUnit test is probably the most > notable example). There are definitely some others written in > kselftest. It's kind of a tough area because integration tests are > kind of defined by being in between unit tests and end-to-end tests. > >>> Test output: >>> ------------ >>> [ 0.000000] Booting Linux on physical CPU 0x0000000100 [0x410fd033] >>> [ 0.000000] Linux version 5.17.0-rc1-next-20220127 >>> (tuxmake@tuxmake) (aarch64-linux-gnu-gcc (Debian 11.2.0-9) 11.2.0, GNU >>> ld (GNU Binutils for Debian) 2.37) #1 SMP PREEMPT @1643255563 >>> [ 0.000000] Machine model: ARM Juno development board (r2) >>> >>> <trimmed output> >>> >>> [ 3.285226] ### dt-test ### start of unittest - you will see error messages >>> [ 3.293269] ### dt-test ### EXPECT \ : Duplicate name in >>> testcase-data, renamed to \"duplicate-name#1\" >>> [ 3.293456] Duplicate name in testcase-data, renamed to \"duplicate-name#1\" >>> [ 3.313367] ### dt-test ### EXPECT / : Duplicate name in >>> testcase-data, renamed to \"duplicate-name#1\" >>> [ 3.314709] ### dt-test ### EXPECT \ : OF: >>> /testcase-data/phandle-tests/consumer-a: could not get >>> #phandle-cells-missing for /testcase-data/phandle-tests/provider1 >>> [ 3.323968] OF: /testcase-data/phandle-tests/consumer-a: could not >>> get #phandle-cells-missing for /testcase-data/phandle-tests/provider1 >>> >>> <trimmed output> >>> >>> [ 5.118400] ### dt-test ### EXPECT / : OF: overlay: ERROR: multiple >>> fragments add and/or delete node >>> /testcase-data-2/substation@100/motor-1/electric >>> [ 5.121358] atkbd serio1: keyboard reset failed on 1c070000.kmi >>> [ 5.134160] ### dt-test ### end of unittest - 257 passed, 0 failed >>> >>> >>> Ref: >>> Full test output of of-unittest >>> https://lkft.validation.linaro.org/scheduler/job/4458582#L1019 >>> https://lkft.validation.linaro.org/scheduler/job/4404330#L428 >>> >>> Kunit example test output that we are running in our daily CI loop. >>> https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.10.y/build/v5.10.70/testrun/5965109/suite/kunit/tests/ >>> >>> Kunit Full test logs: >>> https://lkft.validation.linaro.org/scheduler/job/3643324 >>> >>> https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.10.y/build/v5.10.70/testrun/5965109/suite/kunit/test/kunit_log_test/log >>> >>> >>> -- >>> Linaro LKFT >>> https://lkft.linaro.org >>> >>
