On Thu, May 16, 2024 at 11:54:44AM +0200, Patrick Steinhardt wrote:
> That's certainly the biggest part, yeah. But I have been hitting lots of
> compiler-dependent behaviour. This is mostly in the area of bugs though,
> where for example toolchain A may initialize variables on the stack to
> all zeroes while toolchain B does not.
I've definitely run into differing runtime outcomes for undefined
behavior stuff like that. But in my experience most of that is
consistently found by ASan/UBSan (which we do run in CI these days).
It's possible there are cases that those sanitizers don't catch but that
cause differing behavior. But I can't think of one off the top of my
head where that has happened.
> I'm not sure whether this is sufficient reason on its own to warrant
> testing with several toolchains. But we can easily combine this with
> additional tuning knobs. Two separate test jobs with GCC and Clang are
> comparatively boring. But if we make it GCC+SHA1 and Clang+SHA256 then
> it becomes more interesting.
Yeah. Combining orthogonal properties into a single job lets us cover
both (for the common case of success on both) with less CPU. But:
- it can sometimes be hard to figure out which of the properties was
responsible for a failure. That was the very subject of the thread I
referenced earlier, where "linux-gcc" was "use gcc" and also "set
lots of knobs".
- they might not actually be orthogonal. If you care about checking
runtime behavior in the output of two compilers, then that _could_
manifest only in the sha256 code. Or as you get into more
properties, they may overlap in other ways. I think reftable+sha256
is an interesting (eventual) combo to test on top of reftable+sha1.
-Peff
