On Mon, Jun 22, 2020 at 7:47 PM David Gow <davidgow@xxxxxxxxxx> wrote: > > On Mon, Jun 22, 2020 at 6:45 AM Frank Rowand <frowand.list@xxxxxxxxx> wrote: > > > > Tim Bird started a thread [1] proposing that he document the selftest result > > format used by Linux kernel tests. > > > > [1] https://lore.kernel.org/r/CY4PR13MB1175B804E31E502221BC8163FD830@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > > > The issue of messages generated by the kernel being tested (that are not > > messages directly created by the tests, but are instead triggered as a > > side effect of the test) came up. In this thread, I will call these > > messages "expected messages". Instead of sidetracking that thread with > > a proposal to handle expected messages, I am starting this new thread. > > Thanks for doing this: I think there are quite a few tests which could > benefit from something like this. > > I think there were actually two separate questions: what do we do with > unexpected messages (most of which I expect are useless, but some of > which may end up being related to an unexpected test failure), and how > to have tests "expect" a particular message to appear. I'll stick to > talking about the latter for this thread, but even there there's two > possible interpretations of "expected messages" we probably want to > explicitly distinguish between: a message which must be present for > the test to pass (which I think best fits the "expected message" > name), and a message which the test is likely to produce, but which > shouldn't alter the result (an "ignored message"). I don't see much > use for the latter at present, but if we wanted to do more things with > messages and had some otherwise very verbose tests, it could > potentially be useful. +Dmitry Vyukov, I think you were interested in this for KASAN before we went with the signalling approach. Any thoughts? > The other thing I'd note here is that this proposal seems to be doing > all of the actual message filtering in userspace, which makes a lot of > sense for kselftest tests, but does mean that the kernel can't know if > the test has passed or failed. There's definitely a tradeoff between > trying to put too much needless string parsing in the kernel and > having to have a userland tool determine the test results. The > proposed KCSAN test suite[1] is using tracepoints to do this in the > kernel. It's not the cleanest thing, but there's no reason KUnit or > similar couldn't implement a nicer API around it. > > [1]: https://lkml.org/lkml/2020/6/22/1506 > > > I implemented an API for expected messages that are triggered by tests > > in the Devicetree unittest code, with the expectation that the specific > > details may change when the Devicetree unittest code adapts the KUnit > > API. It seems appropriate to incorporate the concept of expected > > messages in Tim's documentation instead of waiting to address the > > subject when the Devicetree unittest code adapts the KUnit API, since > > Tim's document may become the kernel selftest standard. > > Is having a nice way to handle expected messages the only thing > holding up porting this to KUnit? > > > Instead of creating a very long email containing multiple objects, > > I will reply to this email with a separate reply for each of: > > > > The "expected messages" API implemention and use can be from > > drivers/of/unittest.c in the mainline kernel. > > > > of_unittest_expect - A proof of concept perl program to filter console > > output containing expected messages output > > > > of_unittest_expect is also available by cloning > > https://github.com/frowand/dt_tools.git > > > > An example raw console output with timestamps and expect messages. > > > > An example of console output processed by filter program > > of_unittest_expect to be more human readable. The expected > > messages are not removed, but are flagged. > > > > An example of console output processed by filter program > > of_unittest_expect to be more human readable. The expected > > messages are removed instead of being flagged. > > Cheers, > -- David
