On 22.02.19 17:27, Paolo Bonzini wrote:
> From: Stefan Raspl <raspl@xxxxxxxxxxxxx>
>
> The current tap13 support summarizes testcases on a rather coarse
> granularity. Which sort of defeats the purpose, especially in CI
> environments, where having results for individual tests to pinpoint
> problems helps a lot.
>
> To improve things, this patch makes TAP mode bypass print_result and extract
> results for individual testcases as they are sent to the log file. It then
> introduces test case numbers and writes the plan at the end.
>
> One could argue that post-processing the logs is a rather fragile approach.
> Which is true - but apparently already the case, see e.g. extract_summary
> in scripts/runtime.bash. Plus this is quite cheap, while a proper solution
> would require to modify the kernels, so we can e.g. pass in a counter for
> the test case number. And we would probably have to come up with reasonable
> test case names, while this approach simply derives them from each test
> output.
>
> Before:
> $ ./run_tests.sh -t
> TAP version 13
> ok selftest-setup
> ok intercept
> ok emulator
> ok sieve
> ok sthyi
> ok skey
> ok diag10
> ok pfmf
> ok cmm
> ok vector
> ok gs # SKIP
> ok iep # SKIP
> 1..12
>
> After:
> $ ./run_tests.sh -t
> TAP version 13
> ok 1 - cmm: privileged: Program interrupt: expected(2)
> ok 2 - cmm: invalid ORC 8: Program interrupt: expected(6)
> ok 3 - diag10: lowcore freeing: 0x0000/0x0000: Program interrupt: expected(6)
> ok 4 - diag10: lowcore freeing: 0x1000/0x1000: Program interrupt: expected(6)
> ok 5 - diag10: lowcore freeing: 0x0000/0x1000: Program interrupt: expected(6)
> ok 6 - diag10: start/end: end < start: Program interrupt: expected(6)
> ok 7 - diag10: start/end: unaligned start: Program interrupt: expected(6)
> ok 8 - diag10: start/end: unaligned end: Program interrupt: expected(6)
> ok 9 - diag10: privileged: Program interrupt: expected(2)
> ok 10 - emulator: spm/ipm: cc=0,key=f: bit 32 and 33 set to zero
> ok 11 - emulator: spm/ipm: cc=0,key=f: bit 0-31, 40-63 unchanged
> ok 12 - emulator: spm/ipm: cc=0,key=f: cc and key applied
> ok 13 - emulator: spm/ipm: cc=1,key=9: bit 32 and 33 set to zero
> 1..180
> [...]
>
> Signed-off-by: Stefan Raspl <raspl@xxxxxxxxxxxxx>
> [Almost rewritten, but still giving a lot of credit to Stefan. - Paolo]
> Signed-off-by: Paolo Bonzini <pbonzini@xxxxxxxxxx>
> ---
> run_tests.sh | 59 +++++++++++++++++++++++++++++++++++++++++++---------
> scripts/runtime.bash | 19 +++--------------
> 2 files changed, 52 insertions(+), 26 deletions(-)
>
> diff --git a/run_tests.sh b/run_tests.sh
> index 6b0af19..edfd8a4 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -69,12 +69,50 @@ shift $((OPTIND - 1))
> only_tests="$*"
>
> # RUNTIME_log_file will be configured later
> -RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
> +if [[ $tap_output == "no" ]]; then
> + process_test_output() { cat >> $RUNTIME_log_file; }
> + postprocess_suite_output() { cat; }
> +else
> + process_test_output() {
> + CR=$'\r'
> + while read -r line; do
> + line="${line%$CR}"
> + case "${line:0:4}" in
> + PASS)
> + echo "ok TEST_NUMBER - ${line#??????}" >&3
> + ;;
> + FAIL)
> + echo "not ok TEST_NUMBER - ${line#??????}" >&3
> + ;;
> + SKIP)
> + echo "ok TEST_NUMBER - ${line#??????} # skip" >&3
> + ;;
Nice!
However, you re-use the output of each test in full, e.g.
ok 104 - intercept: stpx: Program interrupt: expected(4) == received(4)
But that's not a stable testcase name: The "received(4)" might change in case of
an error, that's why I was truncating everything from " ==" onwards. And it
would be nice in CI environments to have stable testcase names, makes things a
bit easier when comparing with previous output.
Could you fix that?
Otherwise, thanks for the credit - would be bizarr if I gave a rev'd &
tested-by, though :O
Ciao,
Stefan
