On Fri, Sep 15, 2017 at 03:55:09AM +0000, Bird, Timothy wrote: > > > > -----Original Message----- > > From: Paul on Thursday, September 14, 2017 4:09 PM > > On 08/18/2017 11:57 AM, Shuah Khan wrote: > > > On 08/18/2017 12:48 PM, Bird, Timothy wrote: > > >>> -----Original Message----- > > >>> From: Shuah Khan [mailto:shuahkh@xxxxxxxxxxxxxxx] > > >>> As I am converting tests to TAP13, I am encountering cases where > > >>> a test invokes other tests. For example, say test1 invokes test2 > > >>> and test3, the output will have nested TAP 13 headers and footers. > > >>> > > >>> Example: > > >>> > > >>> TAP version 13 > > >>> > > >>> test1 output > > >>> > > >>> TAP version 13 > > >>> test2 output > > >>> 1..1 > > >>> > > >>> TAP version 13 > > >>> test3 output > > >>> 1..1 > > >>> > > >>> 1..1 > > >>> > > >>> > > >>> This is the case with timers tests. Some tests invoke 3 to 4 tests. > > >>> As these tests can be run independently, they all will need to be > > >>> converted. So we can't avoid nested TAP headers and footers. > > >>> > > >>> Do you see any problems with nesting? > > >> > > >> It's not in the TAP13 protocol itself, but I note that the TAP plugin > > >> for Jenkins (tap4j) supports this kind of nesting. I'd have to do some > > testing > > >> to see exactly how they handle this. For example, the example on the > > >> plugin page shows > > >> nesting similar to what you're doing, but it omits the header line indicating > > >> the TAP protocol used. Your example, by the way, doesn't have any > > >> actual test lines (I assume that "test 2 output" is a placeholder for the > > >> actual test output lines.) > > >> > > >> If one program is calling another program, how is the indentation > > handled? > > >> Would the combined output really look like the above (with subtest > > output > > >> indented?) I believe the Jenkins TAP plugin relies on indentation to > > determine > > >> where the subtest output ends. > > >> > > >> Another question: is the subtest considered a test case of the parent > > test? > > >> That is, does the following output make sense: > > >> > > >> TAP version 13 > > >> # doing timer tests > > >> ok 1 first test > > >> # executing sub-test timer-sub1 > > >> TAP version 13 > > >> ok 1 sub-test one > > >> 1..1 > > >> ok 2 execute sub-test timer-sub1 > > >> ok 3 some other test > > >> 1..3 > > >> ------------------ > > > > > > Yes it does make sense, however, it isn't possible to print > > > the above indented text from the second test, as this test can > > > also be run all by itself. > > What if we added a parameter to the ksft_* output functions that could > > take an indentation level? Then when the parent test executes the sub test > > program, it could provide an int to notify the sub test that it is being run > > as a sub test, and at what sub level (to accommodate sub-sub tests and sub- > > sub-sub > > tests, etc). > > > Or just just pass a string on the command line: > $ someprog --output_indent " " > > Any program that is both called by another program, and calls another > program can just add to the string. Either a string or an int would work. > Yes, I think that's a good idea, and one could even customize the indentation string (not sure why one would want to do that but okay). Although I think that an int might be better to allow for multiple levels of indentation/subtesting. In either case, the ksft_* output function would still have to be notified that it has to print indentation, so the [sub]test would still have to read the command line argument and then assign a variable to plug into each ksft_* call. > I thought about doing something like this. I think only a few programs > would need to take an indentation argument through their command line. > I think it would be a nice way to make it really easy to convert any test program into a sub-test-able test program. > > Then at the beginning of the sub test, it could set a variable that > > represents its sub-level or indentation level, and then plug that variable into > > every ksft_* output call. If it doesn't receive any indentation level then it is > > being run as a top-level test and can then set the indentation level to 0. > > > > I considered a global variable but that's a problem when we get into > > multithreaded > > tests. > > > > > > I know it's not a good idea to change the API, but there are fewer calls to > > replace > > now than later :) > > Maybe we should just add new calls that print with indent, and only use them > where nesting is needed. Again, I suspect nesting is not used widely. > I was thinking that too, though I was concerned that we would end up with lots of small variations of output functions, like ksft_exit_fail() and ksft_exit_fail_msg(). (Although I guess that's the only function with variation, plus we'd only have to change/add ksft_test_result_[pass|fail|skip|error]() ) > I looked at the possibility of using variadic macros to mask the difference in > arguments, but we're already using variadic functions, so this won't work, unless > we get tricky. > What if the macro took the indentation level and the variadic arguments, then executes a printf with the indentation level, then plugs in the variadic arguments to the ksft output function? Or does passing variadic arguments from a macro to a function not work? Paul > -- Tim > -- To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
