[kvm-unit-tests PATCH v2 1/2] tap13: list testcases individually

[Stefan Raspl <raspl@xxxxxxxxxxxxx>]

The current tap13 support summarizes testcases on a rather coarse
granularity. Which sort of defeats the purpose, especially in CI
environments, where having results for individual tests to pinpoint
problems helps a lot. Plus it can be convenient to have the plan in front
of the individual testcases.
This patch improves things by post-processing the log files, extracting
results for individual testcases, introduces test case numbers and puts the
plan in front of the test case results.
One could argue that post-processing the logs is a rather fragile approach.
Which is true - but apparently already the case, see e.g. extract_summary
in scripts/runtime.bash. Plus this is quite cheap, while a proper solution
would require to modify the kernels, so we can e.g. pass in a counter for
the test case number. And we would probably have to come up with reasonable
test case names, while this approach simply derives them from each test
output.

Before:
  $ ./run_tests.sh -t
  TAP version 13
  ok selftest-setup
  ok intercept
  ok emulator
  ok sieve
  ok sthyi
  ok skey
  ok diag10
  ok pfmf
  ok cmm
  ok vector
  ok gs # SKIP
  ok iep # SKIP
  1..12

After:
  $ ./run_tests.sh -t
  TAP version 13
  1..180
  ok 1 - cmm: privileged: Program interrupt: expected(2)
  ok 2 - cmm: invalid ORC 8: Program interrupt: expected(6)
  ok 3 - diag10: lowcore freeing: 0x0000/0x0000: Program interrupt: expected(6)
  ok 4 - diag10: lowcore freeing: 0x1000/0x1000: Program interrupt: expected(6)
  ok 5 - diag10: lowcore freeing: 0x0000/0x1000: Program interrupt: expected(6)
  ok 6 - diag10: start/end: end < start: Program interrupt: expected(6)
  ok 7 - diag10: start/end: unaligned start: Program interrupt: expected(6)
  ok 8 - diag10: start/end: unaligned end: Program interrupt: expected(6)
  ok 9 - diag10: privileged: Program interrupt: expected(2)
  ok 10 - emulator: spm/ipm: cc=0,key=f: bit 32 and 33 set to zero
  ok 11 - emulator: spm/ipm: cc=0,key=f: bit 0-31, 40-63 unchanged
  ok 12 - emulator: spm/ipm: cc=0,key=f: cc and key applied
  ok 13 - emulator: spm/ipm: cc=1,key=9: bit 32 and 33 set to zero
  [...]

Signed-off-by: Stefan Raspl <raspl@xxxxxxxxxxxxx>
---
 run_tests.sh         | 10 +++++++---
 scripts/runtime.bash | 47 +++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 46 insertions(+), 11 deletions(-)

diff --git a/run_tests.sh b/run_tests.sh
index 6b0af19..01acc38 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -84,7 +84,6 @@ function run_task()
 	if [ -z "$testname" ]; then
 		return
 	fi
-	test_number=$((test_number+1))
 
 	while (( $(jobs | wc -l) == $unittest_run_queues )); do
 		# wait for any background test to finish
@@ -110,9 +109,13 @@ mkdir $unittest_log_dir || exit 2
 echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
 
 if [[ $tap_output == "yes" ]]; then
-    test_number=0
+    tap13_lock="`mktemp`"
+    tap13_count="`mktemp`"
+    rm "$tap13_lock"
+    echo 0 >"$tap13_count"
     echo "TAP version 13"
 fi
+
 trap "wait; exit 130" SIGINT
 for_each_unittest $config run_task
 
@@ -120,5 +123,6 @@ for_each_unittest $config run_task
 wait
 
 if [[ $tap_output == "yes" ]]; then
-    echo "1..$test_number"
+    echo "1..`cat "$tap13_count"`"
+    rm $tap13_count
 fi
diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index 2f8026d..f31a80b 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -69,15 +69,44 @@ function print_result()
         fi
         return
     fi
+}
 
-    if [[ $status == "FAIL" ]]; then
-        echo "not ok $testname $reason"
-    elif [[ $status == "PASS" ]]; then
-        echo "ok $testname"
-    elif [[ $status == "SKIP" ]]; then
-        echo "ok $testname # SKIP $reason"
-    else
-        echo "not ok # TODO unknown test status"
+function get_lock () {
+    for (( ;; )); do
+        ln -s "$tap13_count" "$tap13_lock" 2>/dev/null
+        if [ $? -eq 0 ]; then
+            break
+        fi
+        sleep 1
+    done
+}
+
+function generate_tap13_output() {
+    if [[ $tap_output == "yes" ]]; then
+        get_lock
+        test_number=`cat "$tap13_count"`
+        while read -r line; do
+            name="`echo ${line% ==*} | cut -c 7-`"
+            case "${line:0:4}" in
+                PASS)
+                    (( test_number++ ))
+                    echo "ok $test_number - $name"
+                    ;;
+                FAIL)
+                    (( test_number++ ))
+                    echo "not ok $test_number - $name"
+                    ;;
+                SKIP)
+                    (( test_number++ ))
+                    echo "ok $test_number - $name # skip"
+                    ;;
+                *)
+                    continue
+                    ;;
+            esac
+        done < "$unittest_log_dir/$1.log"
+        echo $test_number>"$tap13_count"
+        rm "$tap13_lock"
     fi
 }
 
@@ -161,6 +190,8 @@ function run()
         print_result "FAIL" $testname "$summary"
     fi
 
+    generate_tap13_output "$testname"
+
     return $ret
 }
 
-- 
2.16.4