Hi Ævar, On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote: > On Tue, May 14 2019, Johannes Schindelin wrote: > > > On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote: > > > >> On Tue, May 14 2019, Johannes Schindelin wrote: > >> > >> > What would you think about a mode where random test cases are > >> > skipped? It would have to make sure to provide a way to recreate > >> > the problem, e.g. giving a string that defines exactly which test > >> > cases were skipped. > >> > > >> > I am *sure* that tons of test scripts would fail with that, and we > >> > would probably have to special-case the `setup` "test cases", and > >> > we would have to clean up quite a few scripts to *not* execute > >> > random stuff outside of `test_expect_*`... > >> > >> I think it would be neat, but unrelated to and overkill for spotting > >> the practical problem we have now, which is that we *know* we skip > >> some of this now on some platforms/setups due to prereqs. > > > > I understand, but I am still worried that this is a lot of work for an > > incomplete fix. > > > > For example, the t7600-merge.sh test script that set off this > > conversation has two prereqs that are unmet on Windows: GPG and > > EXECKEEPSPID. On Azure Pipelines' macOS agents, it is only GPG that is > > unmet. So switching off all prereqs would not help macOS with e.g. a > > bug where the GPG test cases are skipped but the EXECKEEPSPID test > > case is not. > > It won't catch such cases, but will catch cases where a later new test > assumes that whatever the state of the test repo it gets is what's > always going to be there. In practice I think that'll catch most such > issues. That's fair. It will also raise awareness of these issues, which should also have a quite beneficial effect. Given that your patch is not even large, I think you're right, it *is* a lot of bang for the buck. Ciao, Dscho
