Re: [kvm-unit-tests PATCH] runtime: indicate failure on crash/timeout/abort in TAP

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 10/03/2022 16.03, Nico Boehr wrote:
When we have crashes, timeouts or aborts, there is currently no indication for
this in the TAP output. When all reports() up to this point succeeded, this
might result in a TAP file looking completely fine even though things went
terribly wrong.

For example, when I set the timeout for the diag288 test on s390x to 1 second,
it fails because it takes quite long, which is properly indicated in the
normal output:

$ ./run_tests.sh diag288
FAIL diag288 (timeout; duration=1s)

But, when I enable TAP output, I get this:

$ ./run_tests.sh -t diag288
TAP version 13
ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
1..6

Which looks like a completely fine TAP file, but actually we ran into a timeout
and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows
something went wrong:

not ok 7 - diag288: timeout; duration=1s

Signed-off-by: Nico Boehr <nrb@xxxxxxxxxxxxx>
---
  scripts/runtime.bash | 12 +++++++++++-
  1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index 6d5fced94246..b41b3d444e27 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -163,9 +163,19 @@ function run()
          print_result "SKIP" $testname "$summary"
      elif [ $ret -eq 124 ]; then
          print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [[ $tap_output != "no" ]]; then

I'd like to avoid "[[" in new code, and the double negation (!= "no) also looks a little bit ugly ... could you please replace this line with:

	if [ "$tap_output" = "yes" ]; then

?

+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
+        fi
      elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame"
+        if [[ $tap_output != "no" ]]; then

dito

+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
      else
+        if [ $ret -eq 127 ] && [[ $tap_output != "no" ]]; then

dito (especially since this mixes [ and [[ in one line)

+            echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
+        fi
          print_result "FAIL" $testname "$summary"
      fi

As Marc already mentioned, it's indeed a little bit sad that we now have parts of the TAP handling in run_tests.sh and some parts in scripts/runtime.bash, but I also can't think of a much nicer solution right now ... so with this cosmetics fixed:

Reviewed-by: Thomas Huth <thuth@xxxxxxxxxx>




[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux