On Thu, Jun 24, 2010 at 18:39, Jakub Narebski <jnareb@xxxxxxxxx> wrote:
> Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> writes:
>
>> TAP, the Test Anything Protocol, is a simple text-based interface
>> between testing modules in a test harness. test-lib.sh's output was
>> already very close to being valid TAP. This change brings it all the
>> way there. Before:
>>
>> $ ./t0005-signals.sh
>> * ok 1: sigchain works
>> * passed all 1 test(s)
>>
>> And after:
>>
>> $ ./t0005-signals.sh
>> ok 1 - sigchain works
>> # passed all 1 test(s)
>> 1..1
>
> This is incomplete information. It only tells us how the output is
> changed for the case when all test_expect_success tests passes. This
> commit message doesn't tell us how failing tests looks like, and how
> test_expect_failure results gets translated.
>
> Take for example a following test script:
> -- >8 --
> #!/bin/sh
>
> test_description='this is a sample test.
>
> This test is here to see various test outputs.'
>
> . ./test-lib.sh
>
> say 'diagnostic message'
>
> test_expect_success 'true test' 'true'
> test_expect_success 'false test' 'false'
>
> test_expect_failure 'true test (todo)' 'true'
> test_expect_failure 'false test (todo)' 'false'
>
> test_debug 'echo "debug message"'
>
> test_done
> -- 8< --
>
> This test script output looks like the following (the comments are not
> part of output, but denote color of given line of test output):
> ----
> * diagnostic message # yellow
> * ok 1: true test
> * FAIL 2: false test # bold red
> false
> * FIXED 3: true test (todo)
> * still broken 4: false test (todo) # bold green
> * fixed 1 known breakage(s) # green
> * still have 1 known breakage(s) # bold red
> * failed 1 among remaining 3 test(s) # bold red
> ----
>
> How would the output of this test look like after TAP-ification?
> Would it still provide color output when run on terminal?
>
> This test also generates summary of test run in a file in
> test-results/ subdirectory. Currently such file can be e.g. named
> test-results/test_test-8030, and consist of
> -- >8 --
> total 4
> success 1
> fixed 1
> broken 1
> failed 1
>
> -- 8< --
>
> Would TAP-ification change that? Even if there is no change, this is
> worth a sentence or a few words in a commit message.
To all of the above: I could be more verbose in the commit message in
needed, but after you raised similar questions in a previous version
of this series I added this to the above commit message:
All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.
I.e. aside from turning "FAIL" into "not ok" and removing the "* "
prefix from lines everything else works exactly as before, that
includes color output and the aggregation in t/test-results/.
I can expand on that paragraph with more examples if you think it's necessary.
Here's how the output from the above test script looks under TAP:
diagnostic message
ok 1 - true test
not ok - 2 false test
# false
ok 3 - true test (todo) # TODO known breakage
not ok 4 - false test (todo) # TODO known breakage
# fixed 1 known breakage(s)
# still have 1 known breakage(s)
# failed 1 among remaining 3 test(s)
1..4
> P.S. With current output pass / fail results of test_expect_failure
> are justified on ':' separator. This is not the case of TAP output.
TAP output is conventionally not justified, unlike the existing
output. I don't know if anything would break if it were, i.e.:
ok 1 - foo
not ok 2 - bar
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html