On 3/16/23 17:59, Rae Moar wrote:
> Add recognition of the test name line ("# Subtest: <name>") to the KTAP v2
> spec.
>
> The purpose of this line is to declare the name of a test before its
> results. This functionality is especially useful when trying to parse test
> results incrementally and when interpretting results after a crash.
>
> This line is already compliant with KTAP v1 as it is interpretted as a
> diagnostic line by parsers. Additionally, the line is currently used by
> KUnit tests and was derived from the TAP 14 spec:
> https://testanything.org/tap-version-14-specification.html.
It is convenient that "# Subtest: <name>" is compatible with v1, but I think
that there is a negative that overrides the convenience.
The "# Subtest: <name>" syntax means that we need to restrict the format of
diagnostic lines, such that "#Subtest:" is an illegal diagnostic, at least
for the line immediately following the Version line.
I think it would be cleaner to modify the Version line syntax to be:
KTAP version 2 [# <subtest_name>]
I notice that the KTAP Specification version 1 fails to specify the
Version line syntax. So the Specification would be updated from:
All KTAP-formatted results begin with a "version line" which specifies which
version of the (K)TAP standard the result is compliant with.
For example:
- "KTAP version 1"
- "TAP version 13"
- "TAP version 14"
to:
The Version line is required and must have the format:
.. code-block:: none
KTAP version 2 [# <subtest_name>]
All KTAP-formatted results begin with a "version line" which specifies which
version of the (K)TAP standard the result is compliant with.
For example:
- "KTAP version 2"
- "TAP version 13"
- "TAP version 14"
>
> Recognition of this line would create an accepted way for different test
> frameworks to declare the name of a test before its results.
>
> The proposed location for this line is between the version line and the
> test plan line. This location ensures that the line would not be
> accidentally parsed as a subtest's diagnostic lines. Note this proposed
> location would be a slight differentiation from KTAP v1.
>
> Example of test name line:
>
> KTAP version 2
> # Subtest: main_test
> 1..1
> KTAP version 2
> # Subtest: sub_test
> 1..2
> ok 1 test_1
> ok 2 test_2
> ok 1 sub_test
>
> Here is a link to a version of the KUnit parser that is able to parse the
> test name line for KTAP version 2. Note this includes a test name line for
> the main level of KTAP.
>
> Link: https://kunit-review.googlesource.com/c/linux/+/5709
>
> Signed-off-by: Rae Moar <rmoar@xxxxxxxxxx>
> ---
>
> This is a RFC. I would like to know what people think and use this as a
> platform for discussion on KTAP v2.
>
> Note: this patch is based on Frank's ktap_spec_version_2 branch.
>
> Documentation/dev-tools/ktap.rst | 19 ++++++++++++++-----
> 1 file changed, 14 insertions(+), 5 deletions(-)
>
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index ff77f4aaa6ef..9c7ed66d9f77 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -28,8 +28,7 @@ KTAP output is built from four different types of lines:
> In general, valid KTAP output should also form valid TAP output, but some
> information, in particular nested test results, may be lost. Also note that
> there is a stagnant draft specification for TAP14, KTAP diverges from this in
> -a couple of places (notably the "Subtest" header), which are described where
> -relevant later in this document.
> +a couple of places, which are described where relevant later in this document.
>
> Version lines
> -------------
> @@ -44,8 +43,8 @@ For example:
> - "TAP version 14"
>
> Note that, in KTAP, subtests also begin with a version line, which denotes the
> -start of the nested test results. This differs from TAP14, which uses a
> -separate "Subtest" line.
^^^^ This is an error in the KTAP Specification version 1. TAP14 allows the case
of "Bare Subtests", which would be the equivalent of the KTAP v1 method.
> +start of the nested test results. This differs from TAP14, which uses only a
> +"Subtest" line.
>
> While, going forward, "KTAP version 2" should be used by compliant tests, it
> is expected that most parsers and other tooling will accept the other versions
> @@ -166,6 +165,12 @@ even if they do not start with a "#": this is to capture any other useful
> kernel output which may help debug the test. It is nevertheless recommended
> that tests always prefix any diagnostic output they have with a "#" character.
>
> +One recognized diagnostic line is the "# Subtest: <name>" line. This line
> +is used to declare the name of a test before subtest results are printed. This
> +is helpful for parsing and for providing context during crashes. As a rule,
> +this line is placed after the version line and before the plan line. Note
> +this line can be used for the main test, as well as subtests.
> +
> Unknown lines
> -------------
>
> @@ -206,6 +211,7 @@ An example of a test with two nested subtests:
> KTAP version 2
> 1..1
> KTAP version 2
> + # Subtest: example
> 1..2
> ok 1 test_1
> not ok 2 test_2
> @@ -219,6 +225,7 @@ An example format with multiple levels of nested testing:
> KTAP version 2
> 1..2
> KTAP version 2
> + # Subtest: example_test_1
> 1..2
> KTAP version 2
> 1..2
> @@ -245,7 +252,7 @@ allows an arbitrary number of tests to be nested no yes
>
> The TAP14 specification does permit nested tests, but instead of using another
> nested version line, uses a line of the form
> -"Subtest: <name>" where <name> is the name of the parent test.
> +"Subtest: <name>" where <name> is the name of the parent test as discussed above.
>
> Example KTAP output
> --------------------
> @@ -254,6 +261,7 @@ Example KTAP output
> KTAP version 2
> 1..1
> KTAP version 2
> + # Subtest: main_test
> 1..3
> KTAP version 2
> 1..1
> @@ -266,6 +274,7 @@ Example KTAP output
> ok 2 test_2
> ok 2 example_test_2
> KTAP version 2
> + # Subtest: example_test_3
> 1..3
> ok 1 test_1
> # test_2: FAIL
>
> base-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5
