On Thu, Mar 24 2022, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> writes: > >> == It "can be parsed" >> == Human readable >> == It's trivial to produce >> == It's more flexible >> == It doesn't matter if nobody else uses it >> == Everyone uses it > > I agree with all of what you said above. > > But do non-human readers really want to read output from an > "--immediate" session? Don't they rather see the whole thing? That test run will still take longer, especially if you're expecting widespread failures. The immediate use-case I have for this is that I have unsubmitted patches to implement a "check" mode for SANITIZE=leak, where it runs with --immediate and "fails" tests that pass under SANITIZE=leak but aren't marked with "TEST_PASSES_SANITIZE_LEAK=true", and passes (and runs to completion completely) those tests that do have "TEST_PASSES_SANITIZE_LEAK=true" but pass with SANITIZE=leak (and fails those that would fail there, no "fail inversion"). That sort of thing is very useful to for any tests where we mark certain other tests as being OK under a given mode, and except there to be a 1=1 correspondance. The reason for the --immediate there is an optimization, it takes a long time to run with SANITIZE=leak, and if any single failure is enough to declare the test bad --immediate is useful. Now, the TAP tooling is quite tolerant of such bad output, another point in its favor over something like JUnit (I'm assuming most libraries using it would pass it through a generic XML parser, whose error handling is either "ok" or "ARGHL!" :). I.e. it'll give you a full parse, but just have a minor complaint about the lacking end marker, getting rid of that output & parse nit is nice. > The only "Huh?" I had with the original patch that started the > thread was that TAP output currently does not work well with the > "--verbose" option, and I've never run our tests with the > "--immediate" option without the "--verbose" option to see where and > how the first breakage happens and what is left behind in the trash > directory, i.e. to bootstrap an interactive debugging session. > > But not being useful for the use case of human reader running > interactively to get the leftover state does not mean improvement > for other use cases is not welcome. I was sad to see that ability go away, without the patches I referenced running through a TAP parser with --verbose works for almost all ouf our test suite, so IMO we were a bit overzelous in adding that early abort for it. But I do have patches to make it pass 100% guaranteed, so depending on your interest ... :) I was planning to queue that behind some of the outstanding test lib changes it would conflict with, e.g. this one & the outstanding CI topics.
