On Fri, Oct 6, 2017 at 12:22 PM, Jeff King <peff@xxxxxxxx> wrote: > On Fri, Oct 06, 2017 at 12:00:05PM -0700, Stefan Beller wrote: > >> The `test_must_fail` should only be used to indicate a git command is >> failing. `test_cmp` is not a git command, such that it doesn't need the >> special treatment of `test_must_fail` (which e.g. includes checking for >> segfault) > > Hmph. "test_must_fail test_cmp" is a weird thing for somebody to write. > And your patch is obviously an improvement, but I have to wonder if some > of these make any sense. Thanks for actually thinking about the problem. I just searched and replaced the combination of test_must_fail and test_cmp mechanically after I noticed one such occurrence whilst preparing the next patch (order of test_cmp args). > > If we're expecting some outcome, then it's reasonable to say: > > 1. The output should look exactly like this. (test_cmp) > > 2. The output should look something like this. (grep) > > 3. The output should _not_ mention this (! grep) > > But "the output should not look exactly like this" doesn't seem very > robust. It's likely to give a false success due to small changes (or > translations), or even bugs in the script. I agree that the case "should not look like exactly this" is most likely indicating a weak test. > Running ./t3504 with "-v" (with or without your patch) shows: > > --- expect 2017-10-06 19:14:43.677840120 +0000 > +++ foo 2017-10-06 19:14:43.705840120 +0000 > @@ -1 +1 @@ > -fatal: cherry-pick: --no-rerere-autoupdate cannot be used with --continue > +foo-dev > > Which just seems like a bug. Did the original author mean foo-expect? This was originally written by a non regular in 2008. I don't think they remember even if we'd ask. I think we'd want to not resolve it to foo-dev here, (so ideally we'd test for <<<< foo ==== foo-dev >>>> but just testing that we do not blindly resolve to foo seems ok-ish) Thanks for spotting this! > It's hard to tell, as we are just reusing expectations from previous > tests. > >> diff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh >> index 02106c9226..7178b917ce 100755 >> --- a/t/t5512-ls-remote.sh >> +++ b/t/t5512-ls-remote.sh >> @@ -56,7 +56,7 @@ test_expect_success 'use "origin" when no remote specified' ' >> >> test_expect_success 'suppress "From <url>" with -q' ' >> git ls-remote -q 2>actual_err && >> - test_must_fail test_cmp exp_err actual_err >> + ! test_cmp exp_err actual_err >> ' > > This one seems like "test_18ngrep ! ^From" would be more appropriate. Or > even "test_must_be_empty". Going by the test title, I agree. >> diff --git a/t/t5612-clone-refspec.sh b/t/t5612-clone-refspec.sh >> index fac5a73851..5f9ad51929 100755 >> --- a/t/t5612-clone-refspec.sh >> +++ b/t/t5612-clone-refspec.sh >> @@ -87,7 +87,7 @@ test_expect_success 'by default no tags will be kept updated' ' >> git for-each-ref refs/tags >../actual >> ) && >> git for-each-ref refs/tags >expect && >> - test_must_fail test_cmp expect actual && >> + ! test_cmp expect actual && >> test_line_count = 2 actual > > Here we check that no updates happened due to a fetch because we see > that the tags in the fetched repo do not match the tags in the parent > repo. That actually seems pretty legitimate. But I think: > > git for-each-ref refs/tags >before > git fetch > git for-each-ref refs/tags >after > test_cmp before after > > would be more straightforward. > >> diff --git a/t/t7508-status.sh b/t/t7508-status.sh >> index 93f162a4f7..1644866571 100755 >> --- a/t/t7508-status.sh >> +++ b/t/t7508-status.sh >> @@ -1532,7 +1532,7 @@ test_expect_success '"status.branch=true" same as "-b"' ' >> test_expect_success '"status.branch=true" different from "--no-branch"' ' >> git status -s --no-branch >expected_nobranch && >> git -c status.branch=true status -s >actual && >> - test_must_fail test_cmp expected_nobranch actual >> + ! test_cmp expected_nobranch actual >> ' > > Shouldn't this be comparing it positively to the output with "--branch"? > >> test_expect_success '"status.branch=true" weaker than "--no-branch"' ' >> diff --git a/t/t9164-git-svn-dcommit-concurrent.sh b/t/t9164-git-svn-dcommit-concurrent.sh >> index d8464d4218..5cd6b40432 100755 >> --- a/t/t9164-git-svn-dcommit-concurrent.sh >> +++ b/t/t9164-git-svn-dcommit-concurrent.sh >> @@ -92,7 +92,7 @@ test_expect_success 'check if post-commit hook creates a concurrent commit' ' >> echo 1 >> file && >> svn_cmd commit -m "changing file" && >> svn_cmd up && >> - test_must_fail test_cmp auto_updated_file au_file_saved >> + ! test_cmp auto_updated_file au_file_saved >> ) >> ' > > This one looked complicated, so I leave it as an exercise for the > reader. :) eh, I was hoping to not stirr up a controversy, but treating this as a drive-by patch "making the tests a better place, one tiny step at a time". Thanks, Stefan > > -Peff
