On Fri, Jul 16 2021, Jeff King wrote:
> On Fri, Jul 16, 2021 at 04:46:12PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
>> > and so on. That requires two new features in test-lib.sh:
>> >
>> > - making a GIT_TEST_RUN variable that is the opposite of GIT_TEST_SKIP
>> > (instead of just the command-line --run).
>> >
>> > - adding GIT_TEST_{RUN,SKIP}_FROM variables to read the values from a
>> > file rather than the environment (I suppose the caller could just
>> > stuff the contents into the variable, but I expect that test-lib.sh
>> > may want to pare down the entries that do not even apply to the
>> > current script for the sake of efficiency in checking each test).
>> >
>> > That infrastructure would then be applicable to other cases, too. Or
>> > even just useful for using another list (or no list at all) when you
>> > are looking at whether other tests are leak-free or not.
>>
>> I've included a mechanism for whitelisting specific globs, the idea was
>> not to have that be too detailed, but we'd e.g. get to the point of t00*
>> or whatever passing.
>>
>> Anything that's a lot more granular than that is doing to suck,
>> e.g. exposing teh GIT_TEST_SKIP and --run features. of specific test
>> numbers, now you need to count your tests if you add one in the middle
>> of one of those, and more likely you won't test under the mode and just
>> see it in CI.
>
> I think you can do the same level of skipping with GIT_TEST_SKIP,
> though. My argument was just that adding a new mechanism does not make
> sense when we already have one. I.e., running:
>
> GIT_SKIP_TESTS='
> t[123456789]*
> t0[^0]*
> t00[^016]*
> t000[469]
> t001[2459]
> t006[0248]
> ' make SANITIZE=leak test
>
> works already to do the same thing. The only thing we might want is a
> nicer syntax (e.g., to allow positive and negative patterns, or to read
> from a file). But that would benefit all users of GIT_SKIP_TESTS, not
> just people interested in leaks.
A glob in this series is t13*config*, you can't do that with
GIT_SKIP_TESTS because it only includes the numeric part of the test,
i.e. t1300, not t1300-config, or t1306-xdg-files.
But sure, it could happen via some other mechanism than the exact one I
picked, or we could add GIT_SKIP_TESTS2 or whatever.
I would like to be able to compile with it and run "make test" without a
wall of failures by default, i.e. we should be able to tell regressions
from known-OK to get anywhere with it, but that's orthagonal to the
exact mechanism.
>> It also means everything works by default, you get an appropriate notice
>> from prove(1), and even if you run one test manually it'll skip, but
>> emit a message saying you can set the env var to force its run.
>
> With GIT_SKIP_TESTS you obviously don't get a message saying "try
> skipping this test" when it fails. :) But IMHO that is not that big a
> deal. You'll get a test failure with good LSan output. If you are
> working on expanding leak-checker coverage, you already know about your
> options for skipping. If you're adding a new test that leaks, you might
> consider fixing the leak (though not always, if it's far from code
> you're touching).
I do think it makes sense as a test mode test-lib.sh is aware of,
e.g. on obvious next step is to not fail everything right away, but just
let the test run and log all failures to a file, then e.g. fail one test
at the end, or if we're running in that mode collate all the callstacks
and emit a summary for the whole test run.
But yes, the message it emits now isn't such a big deal.
