On Tue, Jun 16, 2020 at 4:52 PM Kees Cook <keescook@xxxxxxxxxxxx> wrote: > > On Mon, Jun 15, 2020 at 07:07:34PM +0000, Bird, Tim wrote: > > From: Kees Cook <keescook@xxxxxxxxxxxx> > > > Note: making the plan line required differs from TAP13 and TAP14. I > > > think it's the right choice, but we should be clear. > > > > [...] > > With regards to making it optional or not, I don't have a strong > > preference. The extra info seems helpful in some circumstances. > > I don't know if it's too onerous to make it a requirement or not. > > I'd prefer if it was always there (either at the beginning or the end), > > but if there is some situation where it's quite difficult to calculate, > > then it would be best not to mandate it. I can't think of any impossible > > situations at the moment. > > I think we should require one of: > > - starting plan line > - ending plan line > - ending with something that indicates "I'm done, but I have no idea how > many tests actually ran" (Maybe "1..?") > > To me, the point of the plan line is to be able to say "this test did, > in fact, finish". So even if some test can't even count how many tests > it _ran_, it can at least say "I am now finished". So the counting is actually not the hard part for us, it's figuring out when we have finished. Again, the change that I am working on (I REALLY need to get that out) should fix that, but until we get that upstream, KUnit doesn't actually know when it is done running tests. > > > TAP13/14 makes description optional, are we making it required (I think > > > we should). There seems to be a TAP13/14 "convention" of starting > > > <description> with "- ", which I'm on the fence about it. It does make > > > parsing maybe a little easier. > > > > I would like the description to be required. > > I don't have a strong opinion on the dash. I'm OK with either one (dash > > or no dash), but we should make kselftest and KUnit consistent. > > I find the dash to be distracting -- it doesn't help me scan it, and it > doesn't help a parser (which only needs to find "#"). Yeah, I also prefer spaces and/or "#". I am okay if spaces are optional only to aid human readability. And honestly I don't care about this point too much. Just offering my 2 cents. > > > > Differences between kernel test result format and TAP13: > > > > - in KTAP the "# SKIP" directive is placed after the description on > > > > the test result line > > I sent a bunch of clean-ups for kselftest.h recently[1], but it looks > like we'll need more for adding "description" to skip (right now it only > prints the SKIP reason). > > [1] https://lore.kernel.org/lkml/20200611224028.3275174-1-keescook@xxxxxxxxxxxx/ > > > > Yes Documentation/*.rst Not sure on name yet, but where do kselftest > > > docs live? :) > > Documentation/dev-tools/kselftest.rst > > > > I'll put this at: Documentation/dev-tools/test-results-format.rst > > Sounds good! > > -- > Kees Cook
