Hello everyone, It has been a few months since there has been activity regarding the second version of KTAP. I wanted to bring this topic back up to the surface. Currently, Frank has compiled a list of KTAPv2 patches here at this link: https://elinux.org/Test_Results_Format_Notes#KTAP_version_2 I am interested in starting the process of accepting these patches as changes to the KTAP documentation as KTAPv2. First, a decision that needs to be made is which branch should be used to accept these changes. Frank has a git repository for KTAPv2. However, it is my understanding he has retired. As a reviewer of KUnit, I am happy to take the patches in through the KUnit branch. Would this work for everyone? Second, we need to finalize the changes. A current list of proposed KTAPv2 patches is as follows: [PATCH v3 0/2] begin KTAP spec v2 process [PATCH v3 1/2] ktap_v2: change version to 2-rc in KTAP specification [PATCH v3 2/2] ktap_v2: change "version 1" to "version 2" in examples [KTAP V2 PATCH] ktap_v2: add skip test result [KTAP V2 PATCH v4] ktap_v2: add test metadata Note the patch on adding a skip test result has not yet been reviewed. So please take a look at this change if interested. As a rule, for any feature of KTAPv1 that is replaced in KTAPv2, it will stay in the documentation as allowed but deprecated to allow a smooth transition. If this process sounds good to people, I will try to get discussions going on current patches and ask for reviews. My goal is by the end of the summer, I can apply the approved patches and send them through a chosen branch (potentially KUnit, as discussed above). Let me know what you think. Thanks! -Rae On Sun, Mar 26, 2023 at 7:25 PM Frank Rowand <frowand.list@xxxxxxxxx> wrote: > > In the middle of the thread about a patch to add the skip test result, > I suggested documenting the process of deprecating the KTAP v1 Specification > method of marking a skipped test: > > https://lore.kernel.org/all/490271eb-1429-2217-6e38-837c6e5e328b@xxxxxxxxx/T/#u > > In a reply to that email I suggested that we ought to have a process to transition > the KTAP Specification from v1 to v2, and possibly v3 and future. > > This email is meant to be the root of that discussion. > > My initial thinking is that there are at least three different types of project > and/or community that may have different needs in this area. > > Type 1 - project controls both the test output generation and the test output > parsing tool. Both generation and parsing code are in the same repository > and/or synchronized versions are distributed together. > > Devicetree unittests are an example of Type 1. I plan to maintain changes > of test output to KTAP v2 format in coordination with updating the parser > to process KTAP v2 data. > > Type 2 - project controls both the test output generation and the test output > parsing tool. The test output generation and a parser modifications may be > controlled by the project BUT there are one or more external testing projects > that (1) may have their own parsers, and (2) may have a single framework that > tests multiple versions of the tests. > > I think that kselftest and kunit tests are probably examples of Type 2. I also > think that DT unittests will become a Type 2 project as a result of converting > to KTAP v2 data. > > Type 3 - project may create and maintain some tests, but is primarily a consumer > of tests created by other projects. Type 3 projects typically have a single > framework that is able to execute and process multiple versions of the tests. > > The Fuego test project is an example of Type 3. > > Maybe adding all of this complexity of different Types in my initial thinking > was silly -- maybe everything in this topic is governed by the more complex > Type 3. > > My thinking was that the three different Types of project would be impacted > in different ways by transition plans. Type 3 would be the most impacted, > so I wanted to be sure that any transition plan especially considered their > needs. > > There is an important aspect of the KTAP format that might ease the transition > from one version to another: All KTAP formatted results begin with a "version > line", so as soon as a parser has processed the first line of a test, it can > apply the appropriate KTAP Specification version to all subsequent lines of > test output. A parser implementation could choose to process all versions, > could choose to invoke a version specific parser, or some other approach > all together. > > In the "add skip test results" thread, I suggested deprecating the v1 > method of marking a skipped test in v2, with a scheduled removal of > the v1 method in v3. But since the KTAP format version is available > in the very first line of test output, is it necessary to do a slow > deprecation and removal over two versions? > > One argument to doing a two version deprecation/removal process is that > a parser that is one version older the the test output _might_ be able > to process the test output without error, but would not be able to take > advantage of features added in the newer version of the Specification. > > My opinion is that a two version deprecation/removal process will slow > the Specification update process and lead to more versions of the > Specification over a given time interval. > > A one version deprecation/removal process puts more of a burden on Type 3 > projects and external parsers for Type 2 projects to implement parsers > that can process the newer Specification more quickly and puts a burden > on test maintainers to delay a move to the newer Specification, or possibly > pressure to support selection of more than one Specification version format > for output data. > > One additional item... On the KTAP Specification version 2 process wiki page, > I suggested that it is "desirable for test result parsers that understand the > KTAP Specification version 2 data also be able to parse version 1 data." > With the implication "Converting version 1 compliant data to version 2 compliant > data should not require a "flag day" switch of test result parsers." If this > thread discussion results in a different decision, I will update the wiki. > > Thoughts? > > -Frank > > -- > You received this message because you are subscribed to the Google Groups "KUnit Development" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@xxxxxxxxxxxxxxxx. > To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/6d4afb49-3cb9-f176-61a2-5bbaab698644%40gmail.com.
