On Wed, Jun 29, 2022 at 10:03 AM Taylor Blau <me@xxxxxxxxxxxx> wrote: > > Hi Greg, > > On Wed, Jun 29, 2022 at 09:55:54AM -0700, Gregory Szorc wrote: > > 1. `git repack -A` creates loose objects > > 2. `git maintenance`'s `loose-objects` task deletes those loose objects > > 3. `git repack -A` fails to find the loose objects it just created and > > aborts with `fatal: unable to add recent objects` > > This is a somewhat well-known race that occurs when one Git process > decides unreachable objects are safe to be deleted, but an incoming push > or reference update makes those to-be-deleted objects reachable before > they are actually removed, leaving the repository in a corrupt state. > > I'm surprised that the loose-objects maintenance task deletes those > objects, though, since it just runs `prune-packed` or (the equivalent > of) `repack -d`, neither of which will actually delete objects from the > repository. > > I see that Stolee is already on the CC list, so perhaps he could chime > in on the above. > > In either case, my recommendation would be to keep those unreachable > objects which haven't yet aged out of the repository around for longer, > which will decrease the likelihood of seeing the race. We had to lower gc.pruneExpire from its default of 1 hour because otherwise this would result in the creation of thousands of loose objects. This is normally acceptable. However, on NFS, the churn from lots of file creation and deletion resulted in acceptable performance degradation. We had to lower gc.pruneExpire to minimize the damage. > If your > repository has a lot of unreachable objects (such that storing each > one of them individually as loose is impractical), I would recommend > using cruft packs (e.g., by running either `git repack --cruft -d` or > `git gc --cruft`) to collect those unreachable objects together into a > single pack. > > See Documentation/technical/cruft-packs.txt for more information about > cruft packs. Yes, I saw this new feature in the Git release this week and am hoping to roll it out to better mitigate this general problem! With cruft packs, I anticipate being able to restore a longer gc.pruneExpire value as they should mitigate the NFS performance badness we're working around. Thank you for confirming it should help with the problem.
