On 08/18/2017 11:57 AM, Shuah Khan wrote: > On 08/18/2017 12:48 PM, Bird, Timothy wrote: >>> -----Original Message----- >>> From: Shuah Khan [mailto:shuahkh@xxxxxxxxxxxxxxx] >>> As I am converting tests to TAP13, I am encountering cases where >>> a test invokes other tests. For example, say test1 invokes test2 >>> and test3, the output will have nested TAP 13 headers and footers. >>> >>> Example: >>> >>> TAP version 13 >>> >>> test1 output >>> >>> TAP version 13 >>> test2 output >>> 1..1 >>> >>> TAP version 13 >>> test3 output >>> 1..1 >>> >>> 1..1 >>> >>> >>> This is the case with timers tests. Some tests invoke 3 to 4 tests. >>> As these tests can be run independently, they all will need to be >>> converted. So we can't avoid nested TAP headers and footers. >>> >>> Do you see any problems with nesting? >> >> It's not in the TAP13 protocol itself, but I note that the TAP plugin >> for Jenkins (tap4j) supports this kind of nesting. I'd have to do some testing >> to see exactly how they handle this. For example, the example on the >> plugin page (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.jenkins.io%2Fdisplay%2FJENKINS%2FTAP%2BPlugin&data=01%7C01%7Cpaul.elder%40pitt.edu%7Cc150bd8332b04565625708d4e66b0a18%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1&sdata=NOlxjT5llCD4ff7ZqUv79zP22jND2em%2BU9hPJRyRmMM%3D&reserved=0) shows >> nesting similar to what you're doing, but it omits the header line indicating >> the TAP protocol used. Your example, by the way, doesn't have any >> actual test lines (I assume that "test 2 output" is a placeholder for the >> actual test output lines.) >> >> If one program is calling another program, how is the indentation handled? >> Would the combined output really look like the above (with subtest output >> indented?) I believe the Jenkins TAP plugin relies on indentation to determine >> where the subtest output ends. >> >> Another question: is the subtest considered a test case of the parent test? >> That is, does the following output make sense: >> >> TAP version 13 >> # doing timer tests >> ok 1 first test >> # executing sub-test timer-sub1 >> TAP version 13 >> ok 1 sub-test one >> 1..1 >> ok 2 execute sub-test timer-sub1 >> ok 3 some other test >> 1..3 >> ------------------ > > Yes it does make sense, however, it isn't possible to print > the above indented text from the second test, as this test can > also be run all by itself. What if we added a parameter to the ksft_* output functions that could take an indentation level? Then when the parent test executes the sub test program, it could provide an int to notify the sub test that it is being run as a sub test, and at what sub level (to accommodate sub-sub tests and sub-sub-sub tests, etc). Then at the beginning of the sub test, it could set a variable that represents its sub-level or indentation level, and then plug that variable into every ksft_* output call. If it doesn't receive any indentation level then it is being run as a top-level test and can then set the indentation level to 0. I considered a global variable but that's a problem when we get into multithreaded tests. I know it's not a good idea to change the API, but there are fewer calls to replace now than later :) Paul > >> >> Regarding the overall question of nested TAP test results... >> Personally I have no problem with this. I do note however that this makes the >> TAP protocol not strictly line-oriented. One benefit of TAP is that you can >> parse it very simple using grep. At least, you can parse the test result lines >> themselves, with something like: grep "^ok|^not ok" testlog >> and get counts with 'grep "^ok" testlog | wc -l && grep "^not ok" testlog | wc -l'. >> With nesting, you can still get the lines and global counts, but since test id >> numbers are duplicated, it's a bit trickier to interpret them in this nested >> format. >> >> Of course a more complicated parser would have no problem with nesting. >> I'm OK adding support for nested TAP data to Fuego. I'm guessing since >> the Jenkins TAP plugin supports it, it has been found useful in other circumstances. >> >> We really should document any variations or extensions to TAP that we >> make (allow/recommend). Also, we may want to use a different header >> to indicate a different version of TAP (like "TAP version 13-linux-kernel", >> or "TAP version 13-kselftest", or something like that.) >> >> -- Tim >> >> P.S. can you point me to the timer tests in the kernel you are referring to? >> I assume they're in tools/testing/selftests/timers, but can you point me to some >> that nest? >> > > Please see the following list. change_skew.c is the best example of nesting. > > tools/testing/selftests/timers/change_skew.c: ret = system("./raw_skew"); > tools/testing/selftests/timers/change_skew.c: ret |= system("./inconsistency-check"); > tools/testing/selftests/timers/change_skew.c: ret |= system("./nanosleep"); > tools/testing/selftests/timers/change_skew.c: ret = system("killall -9 ntpd"); > tools/testing/selftests/timers/clocksource-switch.c: fd = open("/sys/devices/system/clocksource/clocksource0/available_clocksource", O_RDONLY); > tools/testing/selftests/timers/clocksource-switch.c: fd = open("/sys/devices/system/clocksource/clocksource0/current_clocksource", O_RDONLY); > tools/testing/selftests/timers/clocksource-switch.c: fd = open("/sys/devices/system/clocksource/clocksource0/current_clocksource", O_WRONLY); > tools/testing/selftests/timers/clocksource-switch.c: ret = system(buf); > tools/testing/selftests/timers/clocksource-switch.c: ret = system("./nanosleep"); > tools/testing/selftests/timers/freq-step.c: * This test checks the response of the system clock to frequency > tools/testing/selftests/timers/set-2038.c: ret = system("date"); > tools/testing/selftests/timers/set-2038.c: ret = system("./inconsistency-check -c 0 -t 20"); > tools/testing/selftests/timers/set-2038.c: ret |= system("./nanosleep"); > tools/testing/selftests/timers/set-2038.c: ret |= system("./nsleep-lat"); > tools/testing/selftests/timers/set-2038.c: /* The rest of the tests can blowup on 32bit systems */ > tools/testing/selftests/timers/skew_consistency.c: return system("./inconsistency-check -c 1 -t 600"); > > thanks, > -- Shuah > -- > To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fvger.kernel.org%2Fmajordomo-info.html&data=01%7C01%7Cpaul.elder%40pitt.edu%7Cc150bd8332b04565625708d4e66b0a18%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1&sdata=skWTz2rVaqGN5F3nmLq4ZjB%2BndnkYn%2FNPSt6LBHI5r0%3D&reserved=0 > -- To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
