On Thu, Nov 3, 2016 at 5:45 AM, Luc Van Oostenryck <luc.vanoostenryck@xxxxxxxxx> wrote: > Such situation may simply show that what was tested is now fixed > and that it's juste the test annotation which need to be adapted, > but can be a sign that something else is broken. > > Reporting the exact result (failure/success, known-to-fail/expect-to-succeed) > make the testsuite more useful and allow to use more efficiently > git-bisect or other automated testing tools. I like what you are doing as a result. But I think the implementation has room to improve. I don't like deep and messy "else" statement. I attach a patch which I modify base on yours. I hope it is easier to read. Care to review it? Chris
--- .pc/9409919-3-5-testsuite-report-as-error-tests-known-to-fail-but-which-succeed.patch/validation/test-suite 2016-10-24 17:37:29.303846000 +0800 +++ validation/test-suite 2016-11-17 23:56:35.404392454 +0800 @@ -161,18 +161,25 @@ do_test() test_failed=1 fi - if [ "$test_failed" -eq "1" ]; then - ko_tests=`expr $ko_tests + 1` - get_tag "check-known-to-fail" $file - if [ "$?" -eq "0" ]; then + get_tag "check-known-to-fail" $file + must_fail=`expr "$?" = 0` + known_ko_tests=`expr $known_ko_tests + $must_fail` + + if [ "$must_fail" -eq "1" ]; then + if [ "$test_failed" -eq "1" ]; then echo "info: test '$file' is known to fail" - known_ko_tests=`expr $known_ko_tests + 1` + else + echo "info: test '$file' is known to fail but succeed!" + test_failed=1 fi - return 1 + fi + + if [ "$test_failed" -eq "1" ]; then + ko_tests=`expr $ko_tests + 1` else ok_tests=`expr $ok_tests + 1` - return 0 fi + return $test_failed } do_test_suite()
