> -----Original Message----- > From: Frank Rowand <frowand.list@xxxxxxxxx> > > On 5/12/22 00:59, Frank Rowand wrote: > > In the middle of the "RFC - kernel test result specification (KTAP)" thread, > > started in August 2021, Tim Bird made a suggestion to allow a prefix to the > > KTAP data format: > > > >> Just as a side note, in some Fuego tests, it was very useful to include an identifier > >> in thethe prefix nested tests. The output looked like this: > >> > >> TAP version 13 > >> 1..2 > >> [batch_id 4] TAP version 13 > >> [batch_id 4] 1..2 > >> [batch_id 4] ok 1 - cyclictest with 1000 cycles > >> [batch_id 4] # problem setting CLOCK_REALTIME > >> [batch_id 4] not ok 2 - cyclictest with CLOCK_REALTIME > >> not ok 1 - check realtime > >> [batch_id 4] TAP version 13 > >> [batch_id 4] 1..1 > >> [batch_id 4] ok 1 - IOZone read/write 4k blocks > >> ok 2 - check I/O performance > >> > >> Can I propose that the prefix not be fixed by the spec, but that the spec indicates that > >> whatever the prefix is on the TAP version line, that prefix must be used with the output for > >> all lines from the test (with the exception of unknown lines)? > > > > The thread was discussing many other items, but this is the one that I want > > to focus on in this new RFC thread. > > > > Tim's original email was: > > > > https://lore.kernel.org/r/BYAPR13MB2503A4B79074D8ED5579345DFDCB9@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > > > There was one reply to this that commented on Tim's suggestion (and also many > > other items in the thread) at: > > > > https://lore.kernel.org/r/202108301226.800F3D6D4@keescook > > > >> Oh, interesting. This would also allow parallel (unique) test execution > >> to be parsable. That sounds workable. (Again, this needs LAVA patching > >> again...) > > > > I found Tim's original suggestion to be useful, so I have come up with > > two possible ways to modify the KTAP specification to implement what Tim > > was thinking about. I would not be surprised if someone else has a better > > suggestion than mine, but I will reply to this email with my two alternatives > > to start a discussion. My alternatives are not in the form of patches, but > > if discussion leads to a good result then I will create a patch for review. > > > > -Frank > > ================================================================================ > Alternative 2 > - Add an optional <prefix string> to test output. > - Add "Configuration info lines", which are used to provide information > about the KTAP format to programs that interpret the KTAP data. The > only type of "Configuration info lines" proposed provides the value > of <prefix string> for programs that process the KTAP output. > - Further types of "Configuration info lines" could be added. > > Alternative 2b > - Add an optional <prefix string> to test output. > - Add a <prefix string> definition line of the form: > ok 0 <prefix string> > This line is essentially a phony "Test case result lines" for test 0. > > ================================================================================ For what it's worth, I much prefer Alternative 1, using the prefix on the KTAP line as the prefix for the lines associated with that test's output. I think it's much simpler. Thanks for following up on this. -- Tim > ## Configuration info lines > > Occur in zero or more test case result lines, where <number> is 0 (zero), > following the "Plan line", before any other "Test case result" line. > > If there is no "Plan line" at the beginning of the test, then the > "Configuration info lines" follow the "Version line", before any > other "Test case result" line. > > format: > > ok 0 <description> # [<directive>] [<diagnostic data>]] > > <result> must be "ok". > > The type of each "Configuration info line" is defined by the value > of <description>. > > Each value of <description> used for a "Configuration info line" must be > listed in the specification. > > Whether # <directive> is optional or required is defined for each type of > "Configuration info line". > > #### Should '[<diagnostic data>]' be included in the format? > > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > # [output] prefix > > ok 0 output_prefix #<prefix string> > > <prefix content is a constant string> > Note: there is no space between "#" and "<prefix string>". If there is one > or more spaces after the "#", then there are part of the <prefix string> > > ## <diagnostic data> must not exist unless there is a way to determine the > ## end of <prefix string>. > ## > ## Adding a way to determine the end of <prefix string> adds much complexity > ## to the consumers of ktap output. For example, if the format was: > ## > ## ok 0 output_prefix #<prefix string>[#<diagnostic data>] > ## > ## then it is not possible for <prefix string> to contain '#', unless > ## a method to escape '#' is provided. E.G.: > ## > ## ok 0 output_prefix #XXX test result \#\#\##<diagnostic data> > ## > ## would result in <prefix string> value of 'XXX test result ###' > ## > ## My recomendation: do not allow <diagnostic data> in output_prefix format. > > All subsequent test lines are prefixed with the <prefix string>. > > Indentation for "Nested tests" follows <prefix string>. The indentation > does NOT precede <prefix string>. > > "Diagnostic lines" are prefixed with the <prefix string>. > > "Unknown lines" may optionally be prefixed with the <prefix string>, but > are not required to be prefixed with the <prefix string>. It is allowed > for some "Unknown lines" to not be prefixed with the <prefix string>, even > if one or more other "Unknown lines" are prefixed with the <prefix string>. > > #### Does prefixing begin immediately (even for a subsequent "Configuration > #### info line" or does it begin with the test 1 "Test case result line" > #### line? > #### > #### This question might be simplified if the output_prefix line is > #### required to be the last "Configuration info line". (Or maybe even if > #### required to be the first "Configuration info line". > > > ================================================================================ > #### discussion notes: > > PRO: > Test output processing programs can detect the value of <prefix string> > more deterministically than Alternative 1. > > CON: > More complex than Alternative 1. (But alternative 2b is less complex than > Alternative 2.) > > ================================================================================
