On Fri, Mar 18 2022, Junio C Hamano wrote:
> Junio C Hamano <gitster@xxxxxxxxx> writes:
>
>> Marking the step that demonstrates the current shortcomings with
>> "MUST FAIL" is a bad taste, but let's pretend that we didn't notice
>> it ;-). Other than that, it looks like an improvement.
>> ...
>>> +test_expect_failure () {
>>> + _test_expect_todo test_expect_failure "$@"
>>> +}
>>> +
>>> +test_expect_todo () {
>>> + _test_expect_todo test_expect_todo "$@"
>>> +}
>>
>> It is a bit surprising that _test_expect_todo does not share more of
>> its implementation with test_expect_success, but let's pretend we
>> didn't see it.
>
> With a bit more tweak, I think this can be made palatable:
>
> * introduce something that is *NOT* test_must_fail but behaves like
> so, but with a bit of magic (see below).
>
> * do not introduce test_expect_todo, but use an improved version of
> test_expect_success.
>
> Let's illustrate what I mean by starting from an example that we
> want to have _after_ fixing an known breakage. Let's say we expect
> that our test preparation succeeds, 'git foo' fails when given a bad
> option, 'git foo' runs successfully, and the command is expected to
> emit certain output. We may assert the ideal future world like so:
>
> test_expect_success 'make sure foo works the way we want' '
> preparatory step &&
> test_must_fail git foo --bad-option >error &&
> grep "expected error message" error &&
> ! grep "unwanted error message" error &&
> git foo >output &&
> grep expected output &&
> ! grep unwanted output
> '
>
> Let's also imagine that right now, option parsing in "git foo",
> works but the main execution of the command does not work.
>
> With test_expect_todo, you have to write something like this
> to document the current breakage:
>
> test_expect_todo 'document breakage' '
> preparatory step &&
> test_must_fail git foo --bad-option >error &&
> grep "expected error message" error &&
> ! grep "unwanted error message" error &&
> test_must_fail git foo >output &&
> ! grep expected output &&
> grep unwanted output
> '
>
> You can see that this makes one thing unclear.
>
> Among the two test_must_fail and two !, which one(s) document the
> breakage? In other words, which one of these four negations do we
> wish to lift eventually? The answer is the latter two (we said that
> handling of "--bad-option" is already working), but it is not obvious
> in the above test_expect_todo test sequence.
>
> I'd suggest we allow our test to be written this way:
>
> test_expect_success 'make sure foo works the way we want' '
> preparatory step &&
> test_must_fail git foo --bad-option >error &&
> grep "expected error message" error &&
> ! grep "unwanted error message" error &&
> test_ki git foo >output &&
> test_ki grep expected output &&
> test_ki ! grep unwanted output
> '
>
> and teach test_expect_success that an invocation of test_ki ("known
> issue"---a better name that is NOT test_must_fail very much welcome)
> means we hope this test someday passes without test_ki but not
> today, i.e. what your test_expect_todo means, and we unfortunately
> have to expect that the lines annotated with test_ki would "fail".
>
> To readers, test_ki should appear as if an annotation to a single
> command that highlights what we want to eventually be able to fix,
> and when the issue around the line marked as test_ki is fixed, we
> can signal the fact by removing it from the beginning of these lines
> (that is why the last one is "test_ki !" and not "! test_ki").
>
> To the test framework and the shell that is running the test,
> test_ki would be almost identical to test_must_fail, i.e. runs the
> rest of the command, catch segv and report, but otherwise the
> failure from that "rest of the command" execution is turned into
> "success" that lets &&-chain to continue. In addition, it tells
> the test_expect_success running it that a success of the test piece
> should be shown as TODO (expected failure).
>
> Hmm?
Have you had the time to look past patch 1/7 of this series? 2/7
introduces a "test_todo" helper, the "test_expect_todo" is just the
basic top-level primitive.
I don't think we can categorically replace all of the
"test_expect_failure" cases, because in some of those it's too much of a
hassle to assert the exact current behavior, or we don't really care.
But I think for most cases instead of a:
test_ki ! grep unwanted output
It makes more sense to do (as that helper does):
test_todo grep --want yay --expect unwanted -- output
Which is quite a handful, which is why the series goes on to
e.g. introduce a todo_test_cmp, used e.g. like this (and we can easily
add more wrappers for common cases):
cat >want <<-EOF &&
$(git rev-parse HEAD)
EOF
cat >expect <<-\EOF &&
error: can't rev-parse stuff
EOF
test_might_fail git some-cmd HEAD >actual &&
todo_test_cmp want expect actual
I.e. if you just want to throw your hands in the air and say "I don't
care how this fails and just emit a 'not ok .. TODO' line" we already
have test_expect_failure for that use-case.
I think in most other cases documenting that something is broken or
should behave differently shouldn't be synonymous with not caring *how*
that unwanted behavior works right now.
Understanding of your past commentary on this topic is that you strongly
objected to not having the test suite output reflect that a given test
was "not ok ... TODO" in some way. I.e. even though I think
"test_expect_success" has the approximate *semantics* we want we
shouldn't use those.
But I think the combination of "test_expect_todo" and the "test_todo"
primitive should satisfy that, and will give us accurate assertions
about what we're actually doing now.
