> -----Original Message----- > From: Rae Moar <rmoar@xxxxxxxxxx> > Add specification for test metadata to the KTAP v2 spec. > > KTAP v1 only specifies the output format of very basic test information: > test result and test name. Any additional test information either gets > added to general diagnostic data or is not included in the output at all. > > The purpose of KTAP metadata is to create a framework to include and > easily identify additional important test information in KTAP. > > KTAP metadata could include any test information that is pertinent for > user interaction before or after the running of the test. For example, > the test file path or the test speed. > > Since this includes a large variety of information, this specification > will recognize notable types of KTAP metadata to ensure consistent format > across test frameworks. See the full list of types in the specification. > > Example of KTAP Metadata: > > KTAP version 2 > # ktap_test: main > # ktap_arch: uml > 1..1 > KTAP version 2 > # ktap_test: suite_1 > # ktap_subsystem: example > # ktap_test_file: lib/test.c > 1..2 > ok 1 test_1 > # ktap_test: test_2 > # ktap_speed: very_slow > # custom_is_flaky: true > ok 2 test_2 > ok 1 test_suite > > The changes to the KTAP specification outline the format, location, and > different types of metadata. > > Here is a link to a version of the KUnit parser that is able to parse test > metadata lines for KTAP version 2. Note this includes test metadata > lines for the main level of KTAP. > > Link: https://kunit-review.googlesource.com/c/linux/+/5889 > > Signed-off-by: Rae Moar <rmoar@xxxxxxxxxx> > --- > Documentation/dev-tools/ktap.rst | 163 ++++++++++++++++++++++++++++++- > 1 file changed, 159 insertions(+), 4 deletions(-) > > diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst > index ff77f4aaa6ef..4480eaf5bbc3 100644 > --- a/Documentation/dev-tools/ktap.rst > +++ b/Documentation/dev-tools/ktap.rst > @@ -17,19 +17,20 @@ KTAP test results describe a series of tests (which may be nested: i.e., test > can have subtests), each of which can contain both diagnostic data -- e.g., log > lines -- and a final result. The test structure and results are > machine-readable, whereas the diagnostic data is unstructured and is there to > -aid human debugging. > +aid human debugging. One exception to this is test metadata lines - a type > +of diagnostic lines. Test metadata is used to identify important supplemental > +test information and can be machine-readable. > > KTAP output is built from four different types of lines: > - Version lines > - Plan lines > - Test case result lines > -- Diagnostic lines > +- Diagnostic lines (including test metadata) > > In general, valid KTAP output should also form valid TAP output, but some > information, in particular nested test results, may be lost. Also note that > there is a stagnant draft specification for TAP14, KTAP diverges from this in > -a couple of places (notably the "Subtest" header), which are described where > -relevant later in this document. > +a couple of places, which are described where relevant later in this document. > > Version lines > ------------- > @@ -166,6 +167,154 @@ even if they do not start with a "#": this is to capture any other useful > kernel output which may help debug the test. It is nevertheless recommended > that tests always prefix any diagnostic output they have with a "#" character. > > +KTAP metadata lines > +------------------- > + > +KTAP metadata lines are a subset of diagnostic lines that are used to include > +and easily identify important supplemental test information in KTAP. > + > +.. code-block:: none > + > + # <prefix>_<metadata type>: <metadata value> > + > +The <prefix> indicates where to find the specification for the type of > +metadata. The metadata types listed below use the prefix "ktap" (See Types of > +KTAP Metadata). > + > +Types that are instead specified by an individual test framework use the > +framework name as the prefix. For example, a metadata type documented by the > +kselftest specification would use the prefix "kselftest". Any metadata type > +that is not listed in a specification must use the prefix "custom". Note the > +prefix must not include spaces or the characters ":" or "_". > + > +The format of <metadata type> and <value> varies based on the type. See the > +individual specification. For "custom" types the <metadata type> can be any > +string excluding ":", spaces, or newline characters and the <value> can be any > +string. > + > +**Location:** > + > +The first KTAP metadata entry for a test must be "# ktap_test: <test name>", > +which acts as a header to associate metadata with the correct test. > + > +For test cases, the location of the metadata is between the prior test result > +line and the current test result line. For test suites, the location of the > +metadata is between the suite's version line and test plan line. See the > +example below. > + > +KTAP metadata for a test does not need to be contiguous. For example, a kernel > +warning or other diagnostic output could interrupt metadata lines. However, it > +is recommended to keep a test's metadata lines together when possible, as this > +improves readability. > + > +**Here is an example of using KTAP metadata:** > + > +:: > + > + KTAP version 2 > + # ktap_test: main > + # ktap_arch: uml > + 1..1 > + KTAP version 2 > + # ktap_test: suite_1 > + # ktap_subsystem: example > + # ktap_test_file: lib/test.c > + 1..2 > + ok 1 test_1 > + # ktap_test: test_2 > + # ktap_speed: very_slow > + # custom_is_flaky: true > + ok 2 test_2 > + # suite_1 passed > + ok 1 suite_1 > + > +In this example, the tests are running on UML. The test suite "suite_1" is part > +of the subsystem "example" and belongs to the file "lib/example_test.c". It has Unless I'm missing something, the file is specified in the example above as list/test.c, not "lib/example_test.c". Other than that, the documentation looks fine to me. -- Tim
