From: Stefan Raspl <raspl@xxxxxxxxxxxxx>
The current tap13 support summarizes testcases on a rather coarse
granularity. Which sort of defeats the purpose, especially in CI
environments, where having results for individual tests to pinpoint
problems helps a lot.
To improve things, this patch makes TAP mode bypass print_result and extract
results for individual testcases as they are sent to the log file. It then
introduces test case numbers and writes the plan at the end.
One could argue that post-processing the logs is a rather fragile approach.
Which is true - but apparently already the case, see e.g. extract_summary
in scripts/runtime.bash. Plus this is quite cheap, while a proper solution
would require to modify the kernels, so we can e.g. pass in a counter for
the test case number. And we would probably have to come up with reasonable
test case names, while this approach simply derives them from each test
output.
Before:
$ ./run_tests.sh -t
TAP version 13
ok selftest-setup
ok intercept
ok emulator
ok sieve
ok sthyi
ok skey
ok diag10
ok pfmf
ok cmm
ok vector
ok gs # SKIP
ok iep # SKIP
1..12
After:
$ ./run_tests.sh -t
TAP version 13
ok 1 - cmm: privileged: Program interrupt: expected(2)
ok 2 - cmm: invalid ORC 8: Program interrupt: expected(6)
ok 3 - diag10: lowcore freeing: 0x0000/0x0000: Program interrupt: expected(6)
ok 4 - diag10: lowcore freeing: 0x1000/0x1000: Program interrupt: expected(6)
ok 5 - diag10: lowcore freeing: 0x0000/0x1000: Program interrupt: expected(6)
ok 6 - diag10: start/end: end < start: Program interrupt: expected(6)
ok 7 - diag10: start/end: unaligned start: Program interrupt: expected(6)
ok 8 - diag10: start/end: unaligned end: Program interrupt: expected(6)
ok 9 - diag10: privileged: Program interrupt: expected(2)
ok 10 - emulator: spm/ipm: cc=0,key=f: bit 32 and 33 set to zero
ok 11 - emulator: spm/ipm: cc=0,key=f: bit 0-31, 40-63 unchanged
ok 12 - emulator: spm/ipm: cc=0,key=f: cc and key applied
ok 13 - emulator: spm/ipm: cc=1,key=9: bit 32 and 33 set to zero
1..180
[...]
Signed-off-by: Stefan Raspl <raspl@xxxxxxxxxxxxx>
[Almost rewritten, but still giving a lot of credit to Stefan. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@xxxxxxxxxx>
---
run_tests.sh | 59 +++++++++++++++++++++++++++++++++++++++++++---------
scripts/runtime.bash | 19 +++--------------
2 files changed, 52 insertions(+), 26 deletions(-)
diff --git a/run_tests.sh b/run_tests.sh
index 6b0af19..edfd8a4 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -69,12 +69,50 @@ shift $((OPTIND - 1))
only_tests="$*"
# RUNTIME_log_file will be configured later
-RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
+if [[ $tap_output == "no" ]]; then
+ process_test_output() { cat >> $RUNTIME_log_file; }
+ postprocess_suite_output() { cat; }
+else
+ process_test_output() {
+ CR=$'\r'
+ while read -r line; do
+ line="${line%$CR}"
+ case "${line:0:4}" in
+ PASS)
+ echo "ok TEST_NUMBER - ${line#??????}" >&3
+ ;;
+ FAIL)
+ echo "not ok TEST_NUMBER - ${line#??????}" >&3
+ ;;
+ SKIP)
+ echo "ok TEST_NUMBER - ${line#??????} # skip" >&3
+ ;;
+ *)
+ ;;
+ esac
+ echo "${line}"
+ done >> $RUNTIME_log_file
+ }
+ postprocess_suite_output() {
+ test_number=0
+ while read -r line; do
+ case "${line}" in
+ ok*|"not ok"*)
+ (( test_number++ ))
+ echo "${line/TEST_NUMBER/${test_number}}" ;;
+ *) echo "${line}" ;;
+ esac
+ done
+ echo "1..$test_number"
+ }
+fi
+
+RUNTIME_log_stderr () { process_test_output; }
RUNTIME_log_stdout () {
if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
- ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
+ ./scripts/pretty_print_stacks.py $1 | process_test_output
else
- cat >> $RUNTIME_log_file
+ process_test_output
fi
}
@@ -84,7 +122,6 @@ function run_task()
if [ -z "$testname" ]; then
return
fi
- test_number=$((test_number+1))
while (( $(jobs | wc -l) == $unittest_run_queues )); do
# wait for any background test to finish
@@ -110,15 +147,17 @@ mkdir $unittest_log_dir || exit 2
echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
if [[ $tap_output == "yes" ]]; then
- test_number=0
echo "TAP version 13"
fi
+
trap "wait; exit 130" SIGINT
-for_each_unittest $config run_task
+
+(
+ # preserve stdout so that process_test_output output can write TAP to it
+ exec 3>&1
+ test "$tap_output" == "yes" && exec > /dev/null
+ for_each_unittest $config run_task
+) | postprocess_suite_output
# wait until all tasks finish
wait
-
-if [[ $tap_output == "yes" ]]; then
- echo "1..$test_number"
-fi
diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index 2f8026d..200d5b6 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -61,23 +61,10 @@ function print_result()
local summary="$3"
local reason="$4"
- if [[ $tap_output == "no" ]]; then
- if [ -z "$reason" ]; then
- echo "`$status` $testname $summary"
- else
- echo "`$status` $testname ($reason)"
- fi
- return
- fi
-
- if [[ $status == "FAIL" ]]; then
- echo "not ok $testname $reason"
- elif [[ $status == "PASS" ]]; then
- echo "ok $testname"
- elif [[ $status == "SKIP" ]]; then
- echo "ok $testname # SKIP $reason"
+ if [ -z "$reason" ]; then
+ echo "`$status` $testname $summary"
else
- echo "not ok # TODO unknown test status"
+ echo "`$status` $testname ($reason)"
fi
}
--
1.8.3.1
