On 09/15/2017 01:17 AM, Paul Elder wrote: > On 09/14/2017 08:55 PM, Bird, Timothy wrote: >> >> >>> -----Original Message----- >>> From: Paul on Thursday, September 14, 2017 4:09 PM >>> On 08/18/2017 11:57 AM, Shuah Khan wrote: >>>> On 08/18/2017 12:48 PM, Bird, Timothy wrote: >>>>>> -----Original Message----- >>>>>> From: Shuah Khan [mailto:shuahkh@xxxxxxxxxxxxxxx] >>>>>> As I am converting tests to TAP13, I am encountering cases where >>>>>> a test invokes other tests. For example, say test1 invokes test2 >>>>>> and test3, the output will have nested TAP 13 headers and footers. >>>>>> >>>>>> Example: >>>>>> >>>>>> TAP version 13 >>>>>> >>>>>> test1 output >>>>>> >>>>>> TAP version 13 >>>>>> test2 output >>>>>> 1..1 >>>>>> >>>>>> TAP version 13 >>>>>> test3 output >>>>>> 1..1 >>>>>> >>>>>> 1..1 >>>>>> >>>>>> >>>>>> This is the case with timers tests. Some tests invoke 3 to 4 tests. >>>>>> As these tests can be run independently, they all will need to be >>>>>> converted. So we can't avoid nested TAP headers and footers. >>>>>> >>>>>> Do you see any problems with nesting? >>>>> >>>>> It's not in the TAP13 protocol itself, but I note that the TAP plugin >>>>> for Jenkins (tap4j) supports this kind of nesting. I'd have to do some >>> testing >>>>> to see exactly how they handle this. For example, the example on the >>>>> plugin page shows >>>>> nesting similar to what you're doing, but it omits the header line indicating >>>>> the TAP protocol used. Your example, by the way, doesn't have any >>>>> actual test lines (I assume that "test 2 output" is a placeholder for the >>>>> actual test output lines.) >>>>> >>>>> If one program is calling another program, how is the indentation >>> handled? >>>>> Would the combined output really look like the above (with subtest >>> output >>>>> indented?) I believe the Jenkins TAP plugin relies on indentation to >>> determine >>>>> where the subtest output ends. >>>>> >>>>> Another question: is the subtest considered a test case of the parent >>> test? >>>>> That is, does the following output make sense: >>>>> >>>>> TAP version 13 >>>>> # doing timer tests >>>>> ok 1 first test >>>>> # executing sub-test timer-sub1 >>>>> TAP version 13 >>>>> ok 1 sub-test one >>>>> 1..1 >>>>> ok 2 execute sub-test timer-sub1 >>>>> ok 3 some other test >>>>> 1..3 >>>>> ------------------ >>>> >>>> Yes it does make sense, however, it isn't possible to print >>>> the above indented text from the second test, as this test can >>>> also be run all by itself. >>> What if we added a parameter to the ksft_* output functions that could >>> take an indentation level? Then when the parent test executes the sub test >>> program, it could provide an int to notify the sub test that it is being run >>> as a sub test, and at what sub level (to accommodate sub-sub tests and sub- >>> sub-sub >>> tests, etc). >> >> >> Or just just pass a string on the command line: >> $ someprog --output_indent " " >> >> Any program that is both called by another program, and calls another >> program can just add to the string. Either a string or an int would work. This would require changing individual tests. We need a solution that can be implemented in the kselftest framework. >> > I think this is a good idea. Then one could customize the indentation string > even (not sure why one would want to do that but okay). Although to allow for > multiple indentation/subtest levels I think an int might be more appropriate. > > In either case, the ksft output functions still have to know how much indentation > to print, so the test program would still have to save the command line argument > and plug it into every output function call. > >> I thought about doing something like this. I think only a few programs >> would need to take an indentation argument through their command line. >> > I think it would be nice way to make it really easy to make other test > programs to sub-test-able as well. > >>> Then at the beginning of the sub test, it could set a variable that >>> represents its sub-level or indentation level, and then plug that variable into >>> every ksft_* output call. If it doesn't receive any indentation level then it is >>> being run as a top-level test and can then set the indentation level to 0. >>> >>> I considered a global variable but that's a problem when we get into >>> multithreaded >>> tests. >>> >> >> >>> I know it's not a good idea to change the API, but there are fewer calls to >>> replace >>> now than later :) >> >> Maybe we should just add new calls that print with indent, and only use them >> where nesting is needed. Again, I suspect nesting is not used widely. Nesting isn't used widely. It is limited to a few tests. timers might be the only one. >> > This might be better than changing the current functions. I was mainly concerned > that we would have a bunch of functions with slightly modified names, like > ksft_exit_fail() and ksft_exit_fail_msg(). Everything that does output (which is > basically everything in kselftest.h) would need a counterpart that does > indentation, indluding ksft_print_header(), ksft_print_cnts(), ksft_print_msg(), > all the ksft_test_result_*, and all the ksft_exit_*. > >> I looked at the possibility of using variadic macros to mask the difference in >> arguments, but we're already using variadic functions, so this won't work, unless >> we get tricky. >> > We could have macros that take in an indentation level int and the variadic arguments, > then does printf with the indentation level, then plugs the variadic arguments into > the ksft output function call. Are we allowed to pass variadic arguments from macro to > function? > Instead of changes to functions or adding a new function, a new environment variable to indicate nesting will meet the needs. There is a need to drive the nesting externally to minimize changes to the tests as well as being able to use it from run_tests target. run_tests target runs all tests in a loop. Shell scripts need coverage and I added it in run_tests in 4.14-rc1. ksft_print_header() could check this env and determine if it has to omit printing the header based on what Tim pointed out in his earlier response that in the case of nesting, header is omitted from sub-levels. Nesting can be a value 1,2,3 etc. Number of tabs can be derived from this. I think I have a good idea on what will work. I will get a patch out in a day or two. I kind of moved this to my back burner with the make O=dir work. thanks, -- Shuah -- To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
