On Thu, Jan 31, 2019 at 09:49:59AM +0100, Uwe Kleine-König wrote: > This simplifies finding the offending test when the build ended with > > KO: out of 584 tests, 527 passed, 57 failed > 56 of them are known to fail > > Signed-off-by: Uwe Kleine-König <uwe@xxxxxxxxxxxxxxxxx> > --- > validation/test-suite | 11 +++++++++-- > 1 file changed, 9 insertions(+), 2 deletions(-) > > diff --git a/validation/test-suite b/validation/test-suite > index f79a9023d95a..a5572d5e6965 100755 > --- a/validation/test-suite > +++ b/validation/test-suite > @@ -403,9 +403,16 @@ do_test() > if [ "$must_fail" -eq "1" ]; then > if [ "$test_failed" -eq "1" ]; then > [ -z "$vquiet" ] && \ > - echo "info: test '$file' is known to fail" > + echo "info: XFAIL: test '$file' is known to fail" > else > - echo "error: test '$file' is known to fail but succeed!" > + echo "error: XPASS: test '$file' is known to fail but succeed!" > + fi > + else > + if [ "$test_failed" -eq "1" ]; then > + echo "error: FAIL: test '$file' is failed" > + else > + [ -z "$vquiet" ] && \ > + echo "info: PASS: test '$file' passed" > fi > fi I can understand the motivation for this but with this the default *visual* output will be even longer and more redundant: 1) The current idea was if a test doesn't output sommething it's that everything was OK. With the patch there will be a 'PASS' line for each of these tests (more than 90% of them). 2) every FAIL tests have already of line displayed with "error: test ... failed" just before the details of the failure are displayed (which means that such a line can be displayed several time for the same test: bad) and the line with FAIL is displayed at the end of the test. 3) in the same message 'XFAIL' & 'is known to fail', 'XPASS' & 'is known to fail but succeed' and 'FAIL' & 'failed' really mean the same (and all of them are grepable); it's just that one seems to for humans and the others for some script. Point 3) doesn't bother me much. Point 2) is easily fixable (both removing the current repetition in case of multiple failure in the same test file and the repetion with FAIL). Point 1) really bothers me. I propose to: *) simply drop the 'PASS' part of the patch *) add a new flag for test-suite (for example --machine-testing) to: -) not display the 'PASS' lines when the flag is not set -) display the FAIL, XFAIL, ... tags only when the flag is set (I suppose that these FAIL, XFAIL, XPASS ... tags will be used or are already used in some concrete automatic system so it wouldn't be a problem to launch the testsuite with a flag). This will keep the current default output (human-)user friendly. Any thoughts? -- Luc
