On 08/18/2017 12:48 PM, Bird, Timothy wrote: >> -----Original Message----- >> From: Shuah Khan [mailto:shuahkh@xxxxxxxxxxxxxxx] >> As I am converting tests to TAP13, I am encountering cases where >> a test invokes other tests. For example, say test1 invokes test2 >> and test3, the output will have nested TAP 13 headers and footers. >> >> Example: >> >> TAP version 13 >> >> test1 output >> >> TAP version 13 >> test2 output >> 1..1 >> >> TAP version 13 >> test3 output >> 1..1 >> >> 1..1 >> >> >> This is the case with timers tests. Some tests invoke 3 to 4 tests. >> As these tests can be run independently, they all will need to be >> converted. So we can't avoid nested TAP headers and footers. >> >> Do you see any problems with nesting? > > It's not in the TAP13 protocol itself, but I note that the TAP plugin > for Jenkins (tap4j) supports this kind of nesting. I'd have to do some testing > to see exactly how they handle this. For example, the example on the > plugin page (https://wiki.jenkins.io/display/JENKINS/TAP+Plugin) shows > nesting similar to what you're doing, but it omits the header line indicating > the TAP protocol used. Your example, by the way, doesn't have any > actual test lines (I assume that "test 2 output" is a placeholder for the > actual test output lines.) > > If one program is calling another program, how is the indentation handled? > Would the combined output really look like the above (with subtest output > indented?) I believe the Jenkins TAP plugin relies on indentation to determine > where the subtest output ends. > > Another question: is the subtest considered a test case of the parent test? > That is, does the following output make sense: > > TAP version 13 > # doing timer tests > ok 1 first test > # executing sub-test timer-sub1 > TAP version 13 > ok 1 sub-test one > 1..1 > ok 2 execute sub-test timer-sub1 > ok 3 some other test > 1..3 > ------------------ Yes it does make sense, however, it isn't possible to print the above indented text from the second test, as this test can also be run all by itself. > > Regarding the overall question of nested TAP test results... > Personally I have no problem with this. I do note however that this makes the > TAP protocol not strictly line-oriented. One benefit of TAP is that you can > parse it very simple using grep. At least, you can parse the test result lines > themselves, with something like: grep "^ok|^not ok" testlog > and get counts with 'grep "^ok" testlog | wc -l && grep "^not ok" testlog | wc -l'. > With nesting, you can still get the lines and global counts, but since test id > numbers are duplicated, it's a bit trickier to interpret them in this nested > format. > > Of course a more complicated parser would have no problem with nesting. > I'm OK adding support for nested TAP data to Fuego. I'm guessing since > the Jenkins TAP plugin supports it, it has been found useful in other circumstances. > > We really should document any variations or extensions to TAP that we > make (allow/recommend). Also, we may want to use a different header > to indicate a different version of TAP (like "TAP version 13-linux-kernel", > or "TAP version 13-kselftest", or something like that.) > > -- Tim > > P.S. can you point me to the timer tests in the kernel you are referring to? > I assume they're in tools/testing/selftests/timers, but can you point me to some > that nest? > Please see the following list. change_skew.c is the best example of nesting. tools/testing/selftests/timers/change_skew.c: ret = system("./raw_skew"); tools/testing/selftests/timers/change_skew.c: ret |= system("./inconsistency-check"); tools/testing/selftests/timers/change_skew.c: ret |= system("./nanosleep"); tools/testing/selftests/timers/change_skew.c: ret = system("killall -9 ntpd"); tools/testing/selftests/timers/clocksource-switch.c: fd = open("/sys/devices/system/clocksource/clocksource0/available_clocksource", O_RDONLY); tools/testing/selftests/timers/clocksource-switch.c: fd = open("/sys/devices/system/clocksource/clocksource0/current_clocksource", O_RDONLY); tools/testing/selftests/timers/clocksource-switch.c: fd = open("/sys/devices/system/clocksource/clocksource0/current_clocksource", O_WRONLY); tools/testing/selftests/timers/clocksource-switch.c: ret = system(buf); tools/testing/selftests/timers/clocksource-switch.c: ret = system("./nanosleep"); tools/testing/selftests/timers/freq-step.c: * This test checks the response of the system clock to frequency tools/testing/selftests/timers/set-2038.c: ret = system("date"); tools/testing/selftests/timers/set-2038.c: ret = system("./inconsistency-check -c 0 -t 20"); tools/testing/selftests/timers/set-2038.c: ret |= system("./nanosleep"); tools/testing/selftests/timers/set-2038.c: ret |= system("./nsleep-lat"); tools/testing/selftests/timers/set-2038.c: /* The rest of the tests can blowup on 32bit systems */ tools/testing/selftests/timers/skew_consistency.c: return system("./inconsistency-check -c 1 -t 600"); thanks, -- Shuah -- To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
