On Wed, May 17, 2017 at 4:45 AM, Junio C Hamano <gitster@xxxxxxxxx> wrote: > Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> writes: > >> Add an optional test to test git-annex. It's guarded by a new >> EXTERNAL_TESTS environment variable. Running this test takes me 10 >> minutes. [Re-arranged your mail because it worked better with my reply] > I do not mind at all to place the simple reproduction recipe Joey > posted as a new test in our test suite, though. That kind of test > that catches changes to externally visible behaviour surely belongs > to our test suite. This is not a replacement for having an isolated test for the issue Joey noted. We should have a separate patch for that, but I did not have time/interest in writing that up. This change is orthagonal to that. > Well, it is one thing to place git-annex under CI to make sure its > latest and greatest works together well with our latest and greatest > (and it may be something we want to see happen), but driving its > tests from our testsuite sounds like a tail wagging the dog, at > least to me. To me this is just a question of: * Is it the case that git-annex tests for a lot of edge cases we don't test for: Yes, probably. As evidenced by them spotting this regression, and not us. * We can (and should) add a test for the specific breakage we caused in 2.13.0, but that's no replacement for other things annex may be covering & we may be missing which'll catch future breakages. * It's a pretty established practice to test a library (git) along with its consumers (e.g. annex) before a major release. * This allows us to do that at minimal cost. I think it makes sense to add this and integration tests for other similar utilities if they're similarly easy to integrate.