Currently, when a test exits prematurely for whatever reason (crash, guest
exception, ...) the test is correctly reported as FAIL by run_tests.sh - even
when all report()s up to that point passed.
The failure is reported by print_result which checks (among others) the exit
code of the arch-specific run script.
But, as soon as one enables the TAP output in run_tests.sh, there is no way to
see that any more, as the print_result is discarded and only the run script's
output is converted to TAP.
External test runners relying on TAP output will thus believe everything is
fine even though we got a crash.
The same also applies to the logfiles.
As a simple fix, have print_result also print to RUNTIME_log_stderr. For each
test, we will then get an additional test line in the TAP which reports the
test's result:
not ok 36 - css: (35 tests, 2 unexpected failures)
The log files will also contain the result:
FAIL: (35 tests, 2 unexpected failures)
This makes it easy to see whether we had a premature exit in the test. The
disadvantage being the number of test lines in the TAP will no longer match the
number of report()s in a test.
Signed-off-by: Nico Boehr <nrb@xxxxxxxxxxxxx>
---
scripts/runtime.bash | 2 ++
1 file changed, 2 insertions(+)
diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index 6d5fced94246..7bb70d50012a 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -60,8 +60,10 @@ function print_result()
if [ -z "$reason" ]; then
echo "`$status` $testname $summary"
+ RUNTIME_log_stderr "$testname" <<< "$status: $summary"
else
echo "`$status` $testname ($reason)"
+ RUNTIME_log_stderr "$testname" <<< "$status: $reason"
fi
}
--
2.31.1
