On Fri, Dec 09 2022, Junio C Hamano wrote:
> Johannes Schindelin <Johannes.Schindelin@xxxxxx> writes:
>
>> Junio, maybe you could clarify your take on this? As project lead, it is
>> your decision to define how Git uses Continuous Builds, and how the
>> project handles failed CI runs.
>
> I have pretty much been with what Peff and Taylor said in the thread
> already ever since we added CMake support to help Windows/VS folks.
> I agree with you that we do not need to run it for Linux or macOS,
> and if the promised/hoped-for benefit, i.e. that running them on
> non-Windows build would uncover issues that are common across the
> platform and help Windows, is not something that is likely to
> materialize, I'd prefer to see our resources (CI time and developer
> attention) not spent on that.
I think this should be addressed by the "I count less than 20 lines in a
~1.1k line recipe that are really "MSVC"-specific" in the sibling
mail[1]. I.e. the large majority of it is generic recipe code that's run
on all platforms.
> I do not think "how the project handles filed CI runs" is a very big
> issue. I often ignore partial failures (e.g. "winVS(n) test job
> triggered rate limit") and the only annoyance I feel is that such a
> temporary failure contribute one more message to my trash mailbox,
> and I can learn to do the same for a test that marked as failed due
> to linux-cmake-ctest job. I expect that regular contributors are
> doing the same pretty much.
>
> How blocking is a CI failure for drive-by contributors who use GGG?
> While I do not necessarily value drive-by contributions as much as
> you do, if such "an unimportant failure we can ignore" discourages
> those coming from GGG route, that would be unfortunate, exactly
> because they may not have contributed anything to the failures.
> This is not just cmake-ctest, but the leak checking job where a new
> use of a tool that is known to be leaky in a test can turn a test
> that has been passing to fail. If we can mark failures in selected
> jobs as non-blocking, we definitely should do so.
I realize that we've been digressing to the larger topic of what to do
with CI in general, but I don't think the question of whether say
"win+VS build" should "soft fail" is something we should conflate with
this "ab/cmake-nix-and-ci" topic.
It doesn't change the status quo there, and I think is a net
improvement.
Even if we make the CI for anything cmake-related soft-fail, this topic
will still help to get it back up to speed, as you'll be able to run the
full cmake+ctest chain on non-Windows.
> Between keeping and marking linux-cmake-ctest as non-blocking, and
> removing it altogether, I am inclined to say that I'd favor the
> latter for the reasons I explained earlier in this message. But to
> help casual contributors coming via GGG, we would anyway need to (1)
> allow submitting even with failing tests, and (2) tell them that it
> is OK to do so. Which means it is not the end of the world, from
> the point of view of helping casual developers, if we had kept these
> brittle CI jobs like linux-cmake-ctest and linux-leaks.
I can peel off the commit that adds the "linux-cmake-ctest" CI job from
this series, or even just make it do the equivalent of:
cmake && make || echo oops, we're broken
So it'll "soft-fail" (AFAIK GitHub CI, unlike GitLab[2] doesn't support
a native way to "soft-fail").
But I don't think that doing that would help without *also* making
"win+VS {build,test}" soft-fail. I.e. if "linux-cmake-ctest" *and*
"win+VS" (with all else passing) you can be pretty sure it's a generic
cmake problem.
If only one or the other is failing somewhere in cmake having the
"linux-cmake-ctest" job now will help narrow down whether it's a
platform-specific cmake issue.
So, just let me know what you'd prefer, but I think per the above even
if you're impatient with cmake failures the "linux-cmake-ctest" job
should help spend less time on them.
1. https://lore.kernel.org/git/221208.86wn726qcv.gmgdl@xxxxxxxxxxxxxxxxxxx/
2. https://docs.gitlab.com/ee/ci/yaml/#allow_failure -- In the UX:
green=passing, red=failing, yellow=soft-fail
