On Fri, Nov 18, 2016 at 12:13:58AM +0800, Christopher Li wrote: > On Thu, Nov 3, 2016 at 5:45 AM, Luc Van Oostenryck > <luc.vanoostenryck@xxxxxxxxx> wrote: > > Such situation may simply show that what was tested is now fixed > > and that it's juste the test annotation which need to be adapted, > > but can be a sign that something else is broken. > > > > Reporting the exact result (failure/success, known-to-fail/expect-to-succeed) > > make the testsuite more useful and allow to use more efficiently > > git-bisect or other automated testing tools. > > I like what you are doing as a result. > > But I think the implementation has room to improve. > I don't like deep and messy "else" statement. > > I attach a patch which I modify base on yours. I hope it is > easier to read. Care to review it? > > Chris Like I said I prefer your version, but could you add the following patch on top of yours? --- From: Luc Van Oostenryck <luc.vanoostenryck@xxxxxxxxx> Date: Sat, 19 Nov 2016 01:28:31 +0100 Subject: [PATCH] testsuite: let the testsuite fail if we got any unexpected result With this patch a failed test but known to fail will still be reported as failed but won't make the whole testsuite to fail. But a test known to fail but which succeed will make the testsuite to fail. Normal tests, the ones not marked as known-to-fail, will of course the testsuite to fail if one of them fail. To put things simply: the testsuite will return a failure if at least one test gave an unexpected result. Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@xxxxxxxxx> --- validation/test-suite | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/validation/test-suite b/validation/test-suite index 4eff0150..ee6a90a5 100755 --- a/validation/test-suite +++ b/validation/test-suite @@ -11,6 +11,10 @@ if [ ! -x "$default_path/sparse-llvm" ]; then disabled_cmds="sparsec sparsei sparse-llvm" fi +# flags: +# - some tests gave an unexpected result +failed=0 + # counts: # - tests that have not been converted to test-suite format # - tests that are disabled @@ -165,6 +169,8 @@ do_test() must_fail=`expr "$?" = 0` known_ko_tests=`expr $known_ko_tests + $must_fail` + [ "$test_failed" -eq "$must_fail" ] || failed=1 + if [ "$must_fail" -eq "1" ]; then if [ "$test_failed" -eq "1" ]; then echo "info: test '$file' is known to fail" @@ -278,5 +284,5 @@ case "$1" in ;; esac -exit 0 +exit $failed -- 2.10.2 -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
