On Fri, May 17, 2024 at 09:59:35AM -0700, Junio C Hamano wrote: > Jeff King <peff@xxxxxxxx> writes: > > > Yeah. Combining orthogonal properties into a single job lets us cover > > both (for the common case of success on both) with less CPU. But: > > > > - it can sometimes be hard to figure out which of the properties was > > responsible for a failure. That was the very subject of the thread I > > referenced earlier, where "linux-gcc" was "use gcc" and also "set > > lots of knobs". > > > > - they might not actually be orthogonal. If you care about checking > > runtime behavior in the output of two compilers, then that _could_ > > manifest only in the sha256 code. Or as you get into more > > properties, they may overlap in other ways. I think reftable+sha256 > > is an interesting (eventual) combo to test on top of reftable+sha1. > > We could consider permuting, then? If we (for the sake of > simplicity) had two jobs available, one compiled with GCC and the > other compiled with clang, we can enumerate other properties > (e.g. <SHA-1 vs SHA-256>, <reftable vs reffiles>) into pairs, and in > one run, GCC may be running SHA-1+reffiles while clang is running > SHA-256+reftable, and in another run, GCC may be running > SHA-256+reffiles, etc. --- eventually we cover all four combinations > (admittedly for different commits). That's a neat idea to get eventual coverage. I have a feeling it would be a pain in practice, though, because now the CI results aren't quite deterministic. So if commit X introduces a bug in some combination, we might not find out until later, and seeing that X passed all tests doesn't absolve it of responsibility anymore. Likewise, I often have to re-run CI to get more data, or to see if a failure is a flake. If it changed what it ran that would be confusing (though I guess you could use the commit hash as the random "seed" for deciding which permutation to run). -Peff
