On Fri, Dec 10, 2021 at 03:38:53AM +0100, Ævar Arnfjörð Bjarmason wrote:
> I just don't think it makes any sense that I edit say refs.[ch], run
> "make test" locally, but only see that something broke in scalar's
> specific use of libgit.a later when I look at GitHub CI.
I'm definitely sympathetic to this. Having been surprised by CI failure
on something that worked locally is annoying at best, and downright
frustrating when you can't easily reproduce the problem.
But isn't that already true for most of the value that CI provides?
While part of its purpose may be a back-stop for folks who don't run
"make test" locally, I think the biggest value is that it covers a much
wider variety of platforms and scenarios that you don't get out of "make
test" already.
In some of those cases you can reproduce the problem locally by tweaking
build or test knobs. But in others it can be quite a bit more
challenging (e.g., something that segfaults only on Windows). At least
in the proposed change here you'd only be a "make test-all" away from
reproducing the problem locally.
I dunno. I don't feel that strongly either way about whether scalar
tests should be part of "make test". Mostly just observing that this is
not exactly a new case.
> If I'm preparing patches for submission I'll need to get CI passing, so
> I'll need to fix those tests & behavior either way as it's
> in-tree. Knowing about the failures later-not-sooner wastes more time,
> not less.
I think there's probably a tradeoff here. How often you get a "late"
notification of a bug (and how much of your time that wastes) versus how
much time you spend locally running tests that you don't care about.
I do agree that CI presents a bit of a conundrum for stuff at the edge
of the project. It's become a de facto requirement for it to pass. In
general that's good. But it means that features which were introduced
under the notion of "the people who care about this area will tend to
its maintenance" slowly become _everybody's_ problem as soon as they
have any CI coverage. Another example here is the cmake stuff. Or the
recent discussion about "-x" and bash.
I wonder if there's a good way to make some CI results informational,
rather than "failing". I.e., run scalar tests via CI, but if you're not
working on scalar, you don't have to care. Folks who are interested in
the area would keep tabs on those results and make sure that Junio's
tree stays passing.
That view disagrees with the final paragraph here, though:
> The reason we do that with the completion is because some changes to
> e.g. tweak getopts will need to have a corresponding change to the
> completion.
>
> The reason we've not done that with contrib/{subtree,mw-to-git}/ is
> because those are thoroughly in the category of only incidentally being
> in-tree.
> [...]
> Scalar is thoroughly on the "completion" side of that divide, not
> "subtree".
I haven't followed the discussion closely, but in my mind "scalar" was
still in the "it may live in-tree for convenience, but people who aren't
working on it don't necessarily need to care about it" camp. Maybe
that's not the plan, though.
-Peff
