Johannes Schindelin <Johannes.Schindelin@xxxxxx> writes: > In particular, the tests t2024 and t5552 are broken for > ma/doc-diff-usage-fix on Windows. The reason seems to be that those are > already broken for the base commit that Junio picked: > jk/diff-rendered-docs (actually, not the tip of it, but the commit fixed > by Martin's patch). > > Of course, I understand why Junio picks base commits that are far, far in > the past (and have regressions that current `master` does not have), it > makes sense from the point of view where the fixes should be as close to > the commits they fix. The downside is that we cannot automate regression > testing more than we do now, with e.g. myself acting as curator of test > failures. I think by "regressions that current master does not have", you actually meant "unrelated known breakages we have fixed since then". If you use topic-branch workflow, it is unavoidable and expected, as each topic has its focus and the topic to fix doc-diff are not about making checkout test or fetch negotiator test to pass on all platforms. I am wondering if the automated testing can be made more useful by limiting the scope of testing if it is run on individual topic. For four primary integration branches, we do want to ensure all tests keep passing (or at least no tests that passed in the previous round start failing), but it would match the topic-branch workflow better if the automated testing allowed an individual topic to rely on unrelated known breakages to be dealt with by other topics and newer integration branches. For a topic like doc-diff that is primarily meant for developers and documenters, it does not matter much, but for an old but important bug, forking the topic to fix it at a point close to the origin is crucial---that is what would allow people to merge the fix to an older maintenance track, without cherry-picking. It is especially true when the bug being fixed is more severe than unrelated breakages that have been fixed since then. Possible approaches I considered and rejected are: - Perhaps find updated tests in the topic and run only those? We can complain a topic that is meant as a "fix" to something if it does not have test while trying to find such tests ;-) But this does not work if a "fix" has unintended consequences and breaks existing tests that do not have apparent relation known to the author of the patch. - Perhaps find the fork point, run tests to find known breakages and exclude them? This would simply be not practical, as it doubles the number of tests run, for individual topic branches because there are an order of magnitude more of them than the primary integration branches. Possibly, if we limit the fork point to tagged releases, the latter approach might turn out to be doable. For ma/doc-diff-usage-fix, the base commit ad51743007 was v2.20.0-rc0~232, so we could round up and pick v2.20.0 as the base instead (the purpose of picking an older point is to allow merging to older maintenance track, and it is unlikely that people would keep using v2.20.0-rc0 as the base of their forked distribution). Then if the automated testing keeps record of known breakages (whether they are later fixed or still broken) for these tagged releases, testing a new topic forked from one of them and deciding not to worry about breakages of tests that are known to be broken (or perhaps deciding to skip these tests in the first place) may become feasible. I dunno. Limiting fork points to tagged releases for topics that are meant for the maintenance tracks may have other benefits by restricting the shape of the resulting dag, so I am willing to try it as an experiment, whether it would help your workflow or not.
