> -----Original Message----- > From: Shuah Khan [mailto:shuahkh@xxxxxxxxxxxxxxx] > As I am converting tests to TAP13, I am encountering cases where > a test invokes other tests. For example, say test1 invokes test2 > and test3, the output will have nested TAP 13 headers and footers. > > Example: > > TAP version 13 > > test1 output > > TAP version 13 > test2 output > 1..1 > > TAP version 13 > test3 output > 1..1 > > 1..1 > > > This is the case with timers tests. Some tests invoke 3 to 4 tests. > As these tests can be run independently, they all will need to be > converted. So we can't avoid nested TAP headers and footers. > > Do you see any problems with nesting? It's not in the TAP13 protocol itself, but I note that the TAP plugin for Jenkins (tap4j) supports this kind of nesting. I'd have to do some testing to see exactly how they handle this. For example, the example on the plugin page (https://wiki.jenkins.io/display/JENKINS/TAP+Plugin) shows nesting similar to what you're doing, but it omits the header line indicating the TAP protocol used. Your example, by the way, doesn't have any actual test lines (I assume that "test 2 output" is a placeholder for the actual test output lines.) If one program is calling another program, how is the indentation handled? Would the combined output really look like the above (with subtest output indented?) I believe the Jenkins TAP plugin relies on indentation to determine where the subtest output ends. Another question: is the subtest considered a test case of the parent test? That is, does the following output make sense: TAP version 13 # doing timer tests ok 1 first test # executing sub-test timer-sub1 TAP version 13 ok 1 sub-test one 1..1 ok 2 execute sub-test timer-sub1 ok 3 some other test 1..3 ------------------ Regarding the overall question of nested TAP test results... Personally I have no problem with this. I do note however that this makes the TAP protocol not strictly line-oriented. One benefit of TAP is that you can parse it very simple using grep. At least, you can parse the test result lines themselves, with something like: grep "^ok|^not ok" testlog and get counts with 'grep "^ok" testlog | wc -l && grep "^not ok" testlog | wc -l'. With nesting, you can still get the lines and global counts, but since test id numbers are duplicated, it's a bit trickier to interpret them in this nested format. Of course a more complicated parser would have no problem with nesting. I'm OK adding support for nested TAP data to Fuego. I'm guessing since the Jenkins TAP plugin supports it, it has been found useful in other circumstances. We really should document any variations or extensions to TAP that we make (allow/recommend). Also, we may want to use a different header to indicate a different version of TAP (like "TAP version 13-linux-kernel", or "TAP version 13-kselftest", or something like that.) -- Tim P.S. can you point me to the timer tests in the kernel you are referring to? I assume they're in tools/testing/selftests/timers, but can you point me to some that nest? ��.n��������+%������w��{.n�����{��K ����n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
