Jeff King <peff@xxxxxxxx> writes: > Yeah. Combining orthogonal properties into a single job lets us cover > both (for the common case of success on both) with less CPU. But: > > - it can sometimes be hard to figure out which of the properties was > responsible for a failure. That was the very subject of the thread I > referenced earlier, where "linux-gcc" was "use gcc" and also "set > lots of knobs". > > - they might not actually be orthogonal. If you care about checking > runtime behavior in the output of two compilers, then that _could_ > manifest only in the sha256 code. Or as you get into more > properties, they may overlap in other ways. I think reftable+sha256 > is an interesting (eventual) combo to test on top of reftable+sha1. We could consider permuting, then? If we (for the sake of simplicity) had two jobs available, one compiled with GCC and the other compiled with clang, we can enumerate other properties (e.g. <SHA-1 vs SHA-256>, <reftable vs reffiles>) into pairs, and in one run, GCC may be running SHA-1+reffiles while clang is running SHA-256+reftable, and in another run, GCC may be running SHA-256+reffiles, etc. --- eventually we cover all four combinations (admittedly for different commits).
