On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:
>
> On Mon, Oct 31 2022, Mark Hills wrote:
>
> > Our use case: we commit some compiled objects to the repo, where compiling
> > is either slow or requires software which is not always available.
> >
> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade)
> > we are noticing a change in build behaviour.
> >
> > Now, after a "git clone" we find the Makefile intermittently attempting
> > (and failing) some builds that are not intended.
> >
> > Indeed, Make is acting reasonably as the source file is sometimes
> > marginally newer than the destination (both checked out by Git), example
> > below.
> >
> > I've never had to consider consistency timestamps within a Git checkout
> > until now.
> >
> > It's entirely possible there's _never_ a guarantee of consistency here.
> >
> > But then something has certainly changed in practice, as this fault has
> > gone from never happening to now every couple of days.
> >
> > Imaginging I can't be the first person to encounter this, I searched for
> > existing threads or docs, but overwhemingly the results were question of
> > Git tracking the timestamps (as part of the commit) which this is not;
> > it's consistency within one checkout.
> >
> > $ git clone --depth 1 file:///path/to/repo.git
> >
> > $ stat winner.jpeg
> > File: winner.jpeg
> > Size: 258243 Blocks: 520 IO Block: 4096 regular file
> > Device: fd07h/64775d Inode: 33696 Links: 1
> > Access: (0644/-rw-r--r--) Uid: ( 106/ luthier) Gid: ( 106/ luthier)
> > Access: 2022-10-31 16:05:17.756858496 +0000
> > Modify: 2022-10-31 16:05:17.756858496 +0000
> > Change: 2022-10-31 16:05:17.756858496 +0000
> > Birth: -
> >
> > $ stat winner.svg
> > File: winner.svg
> > Size: 52685 Blocks: 112 IO Block: 4096 regular file
> > Device: fd07h/64775d Inode: 33697 Links: 1
> > Access: (0644/-rw-r--r--) Uid: ( 106/ luthier) Gid: ( 106/ luthier)
> > Access: 2022-10-31 16:05:17.766859030 +0000
> > Modify: 2022-10-31 16:05:17.766859030 +0000
> > Change: 2022-10-31 16:05:17.766859030 +0000
> > Birth: -
> >
> > Elsewhere in the repository, it's clear the timestamps are not consistent:
> >
> > $ stat Makefile
> > File: Makefile
> > Size: 8369 Blocks: 24 IO Block: 4096 regular file
> > Device: fd07h/64775d Inode: 33655 Links: 1
> > Access: (0644/-rw-r--r--) Uid: ( 106/ luthier) Gid: ( 106/ luthier)
> > Access: 2022-10-31 16:05:51.628660212 +0000
> > Modify: 2022-10-31 16:05:17.746857963 +0000
> > Change: 2022-10-31 16:05:17.746857963 +0000
> > Birth: -
>
> I think you're almost certainly running into the parallel checkout,
> which is new in that revision range. Try tweaking checkout.workers and
> checkout.thresholdForParallelism (see "man git-config").
Thanks, it will be interesting to try this and I'll report back.
> I can't say without looking at the code/Makefile (and even then, I don't
> have time to dig here:), but if I had to bet I'd say that your
> dependencies have probably always been broken with these checked-in
> files, but they happend to work out if they were checked out in sorted
> order.
>
> And now with the parallel checkout they're not guaranteed to do that, as
> some workers will "race ahead" and finish in an unpredictable order.
These are very simple Makefile rules, I don't think these dependencies are
broken; but your theory is in good alignment with the observed behaviour.
For example, the rule from the recent case above is:
%.jpeg: %.png
convert $< $(IMFLAGS) $@
%.png: %.svg
inkscape --export-type=png --export-filename=$@ $<
As you suggest, perhaps the Git implementation previously ran checked out
in some kind of time order then this happens to fulfil a useful behaviour.
Specificaly with build artefacts. These are likely to have been added to
the repo after the source file. This could have been providing some
pratical and useful tendency of ordering.
> But that's all just a guess, perhaps it has nothing to do with parallel
> checkout, such dependency issues are sensitive to all sorts of other
> things, e.g. maybe git got slightly faster (or slower), so now files
> that were always on different seconds (or the same) aren't in the state
> they were in before...
Hopefully I'll get to some experiments to narrow this down.
Thanks
--
Mark
