On Sat, Jan 23, 2021 at 09:41:29PM -0500, Philippe Blain wrote:
> > So we could split the macos test out. It would probably involve
> > duplicating a little bit of the content, but we do something similar for
> > the dockerized builds. It might be that there is an option we can set to
> > say "keep building the others even if one of these fails", which would
> > give us the best of both.
>
> Yes, a quick Google search pointed me to a blog post [1]
> that mentions using 'fail-fast: false' in the test matrix so that
> one failing job does not automatically cancel the rest of the jobs
> in the matrix (the default is 'true') [2].
>
> If we apply that to all four matrices in the workflow file,
> (windows-test, vs-test, regular and dockerized), it would be
> something like this:
Thanks, that's exactly what I think we'd want.
The downside, of course, is that a failure that will happen on every
platform will mean wasted CPU to trigger the same failure over and over.
But:
- I rarely see that myself, because I wouldn't bother pushing up to CI
until "make test" passed locally. So usually I'm finding portability
issues via CI. Other people might be different, though.
- we already have the Windows tests in a separate matrix anyway, so a
failure on Linux would run the whole Windows suite (which is an
order of magnitude more expensive)
- even within the Windows matrix, I think running the rest of the
tests after a failure is still valuable. If there's a second
failure, you save a round-trip to CI (so it doesn't reduce CPU, but
it may help latency to reach a passing state).
-Peff
