On Tue, Mar 09, 2021 at 10:03:54PM +0100, Ævar Arnfjörð Bjarmason wrote: > > On Tue, Mar 09 2021, SZEDER Gábor wrote: > > > On Tue, Mar 09, 2021 at 05:02:12PM +0100, Ævar Arnfjörð Bjarmason wrote: > >> The eventual goal not included in this series is to have multiple > >> output targets, and e.g. convert ci/print-test-failures.sh to use a > >> TAP parser. > >> > >> Machine-readable "TAP --verbose -x" output can bring us a lot of nice > >> things down the line, I have some local WIP code that's a smarter > >> version of ci/print-test-failures.sh that knows how to spew out only > >> the output relevant to the failing test(s). > > > > I wonder what you mean by the word "relevant" here, as I can't imagine > > how you could possibly identify what is relevant for a failing test > > and what isn't. If you didn't at all meant "relevant", but that it > > will show only the output of the failing test(s), then this is a bad > > example. Our test cases depend too much on previous test cases, and a > > failure of one test can be the result of a change in a previous > > successful tests. > > > > Therefore, any such change to 'ci/print-test-failures.sh' will have my > > firm NACK. > > On e.g. the github CI every step in the run is an collapsable button, so > we could have our cake and eat it too here. Well, one of the things I didn't like in GitHub CI is that I had to click a lot to get to the information I wanted... > It seems to me like a sane default would be to have an equivalent to > "print-test-failures.sh" that only prints the --verbose output for the > failing tests, and a "print-all-output-for-failing-tests.sh" or whatever > which gave you the full output. > > Even then, there seem to me to be some low hanging fruit for > abbreviating even that output. E.g. if you we have 100 tests and we > failed only on the 5th, isn't the --verbose -x output up to and > including the 5th going to be enough, or do we need it for the other 95? I have a test fix waiting to be sent out, where a failing git command in the 'test_when_finished' block of test #21 went unnoticed because of the broken && chain, and ultimately caused the failure of test #91. In my opinion your proposed changes to 'ci/print-test-failures.sh' would make things worse.
