On Wed, Jan 31, 2024 at 5:22 PM Bird, Tim <Tim.Bird@xxxxxxxx> wrote: > > > > > -----Original Message----- > > From: Rae Moar <rmoar@xxxxxxxxxx> > > Add specification for test metadata to the KTAP v2 spec. > > > > KTAP v1 only specifies the output format of very basic test information: > > test result and test name. Any additional test information either gets > > added to general diagnostic data or is not included in the output at all. > > > > The purpose of KTAP metadata is to create a framework to include and > > easily identify additional important test information in KTAP. > > > > KTAP metadata could include any test information that is pertinent for > > user interaction before or after the running of the test. For example, > > the test file path or the test speed. > > > > Since this includes a large variety of information, this specification > > will recognize notable types of KTAP metadata to ensure consistent format > > across test frameworks. See the full list of types in the specification. > > > > Example of KTAP Metadata: > > > > KTAP version 2 > > # ktap_test: main > > # ktap_arch: uml > > 1..1 > > KTAP version 2 > > # ktap_test: suite_1 > > # ktap_subsystem: example > > # ktap_test_file: lib/test.c > > 1..2 > > ok 1 test_1 > > # ktap_test: test_2 > > # ktap_speed: very_slow > > # custom_is_flaky: true > > ok 2 test_2 > > ok 1 test_suite > > > > The changes to the KTAP specification outline the format, location, and > > different types of metadata. > > > > Here is a link to a version of the KUnit parser that is able to parse test > > metadata lines for KTAP version 2. Note this includes test metadata > > lines for the main level of KTAP. > > > > Link: https://kunit-review.googlesource.com/c/linux/+/5889 > > > > Signed-off-by: Rae Moar <rmoar@xxxxxxxxxx> > > --- > > Documentation/dev-tools/ktap.rst | 163 ++++++++++++++++++++++++++++++- > > 1 file changed, 159 insertions(+), 4 deletions(-) > > > > diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst > > index ff77f4aaa6ef..4480eaf5bbc3 100644 > > --- a/Documentation/dev-tools/ktap.rst > > +++ b/Documentation/dev-tools/ktap.rst > > @@ -17,19 +17,20 @@ KTAP test results describe a series of tests (which may be nested: i.e., test > > can have subtests), each of which can contain both diagnostic data -- e.g., log > > lines -- and a final result. The test structure and results are > > machine-readable, whereas the diagnostic data is unstructured and is there to > > -aid human debugging. > > +aid human debugging. One exception to this is test metadata lines - a type > > +of diagnostic lines. Test metadata is used to identify important supplemental > > +test information and can be machine-readable. > > > > KTAP output is built from four different types of lines: > > - Version lines > > - Plan lines > > - Test case result lines > > -- Diagnostic lines > > +- Diagnostic lines (including test metadata) > > > > In general, valid KTAP output should also form valid TAP output, but some > > information, in particular nested test results, may be lost. Also note that > > there is a stagnant draft specification for TAP14, KTAP diverges from this in > > -a couple of places (notably the "Subtest" header), which are described where > > -relevant later in this document. > > +a couple of places, which are described where relevant later in this document. > > > > Version lines > > ------------- > > @@ -166,6 +167,154 @@ even if they do not start with a "#": this is to capture any other useful > > kernel output which may help debug the test. It is nevertheless recommended > > that tests always prefix any diagnostic output they have with a "#" character. > > > > +KTAP metadata lines > > +------------------- > > + > > +KTAP metadata lines are a subset of diagnostic lines that are used to include > > +and easily identify important supplemental test information in KTAP. > > + > > +.. code-block:: none > > + > > + # <prefix>_<metadata type>: <metadata value> > > + > > +The <prefix> indicates where to find the specification for the type of > > +metadata. The metadata types listed below use the prefix "ktap" (See Types of > > +KTAP Metadata). > > + > > +Types that are instead specified by an individual test framework use the > > +framework name as the prefix. For example, a metadata type documented by the > > +kselftest specification would use the prefix "kselftest". Any metadata type > > +that is not listed in a specification must use the prefix "custom". Note the > > +prefix must not include spaces or the characters ":" or "_". > > + > > +The format of <metadata type> and <value> varies based on the type. See the > > +individual specification. For "custom" types the <metadata type> can be any > > +string excluding ":", spaces, or newline characters and the <value> can be any > > +string. > > + > > +**Location:** > > + > > +The first KTAP metadata entry for a test must be "# ktap_test: <test name>", > > +which acts as a header to associate metadata with the correct test. > > + > > +For test cases, the location of the metadata is between the prior test result > > +line and the current test result line. For test suites, the location of the > > +metadata is between the suite's version line and test plan line. See the > > +example below. > > + > > +KTAP metadata for a test does not need to be contiguous. For example, a kernel > > +warning or other diagnostic output could interrupt metadata lines. However, it > > +is recommended to keep a test's metadata lines together when possible, as this > > +improves readability. > > + > > +**Here is an example of using KTAP metadata:** > > + > > +:: > > + > > + KTAP version 2 > > + # ktap_test: main > > + # ktap_arch: uml > > + 1..1 > > + KTAP version 2 > > + # ktap_test: suite_1 > > + # ktap_subsystem: example > > + # ktap_test_file: lib/test.c > > + 1..2 > > + ok 1 test_1 > > + # ktap_test: test_2 > > + # ktap_speed: very_slow > > + # custom_is_flaky: true > > + ok 2 test_2 > > + # suite_1 passed > > + ok 1 suite_1 > > + > > +In this example, the tests are running on UML. The test suite "suite_1" is part > > +of the subsystem "example" and belongs to the file "lib/example_test.c". It has > Unless I'm missing something, the file is specified in the example above as > list/test.c, not "lib/example_test.c". > > Other than that, the documentation looks fine to me. > -- Tim > Hello! Thanks for looking over the documentation. And sorry about that typo in the explanation. It should say "belongs to the file "lib/test.c"". I will change that for the next version. Thanks! -Rae >
