> On Mar 16, 2020, at 6:10 AM, Derrick Stolee <stolee@xxxxxxxxx> wrote: > > On 3/16/2020 4:23 AM, Andreas Krey wrote: >> Hi all, >> >> we occasionally seeing things like this: >> >> | DEBUG: 11:25:20: git -c advice.fetchShowForcedUpdates=false fetch --no-show-forced-updates -q --prune > > I'm happy to see these options. I hope they are helping you! > >> | Warning: Permanently added '[socgit.$company.com]:7999' (RSA) to the list of known hosts. >> | remote: fatal: packfile ./objects/pack/pack-20256f2be3bd51b57e519a9f2a4d3df09f231952.pack cannot be accessed > This _could_ mean a lot of things, but.... > >> | error: git upload-pack: git-pack-objects died with error. >> | fatal: git upload-pack: aborting due to possible repository corruption on the remote side. >> | remote: aborting due to possible repository corruption on the remote side. >> | fatal: protocol error: bad pack header >> >> and when you look in the server repository there is a new packfile dated just around >> that time. It looks like the fetch tries to access a packfile that it assumes to exist, >> but the GC on the server throws it away just in that moment, and thus upload-pack fails. > > ...your intuition about repacking seems accurate. The important part of the > race condition is likely that the server process read and holds a read handle > on the .idx file, but when looking for the object contents it tries to open > the .pack file which was deleted. > [snip] > >> Is there a way to avoid this? >> >> Should there be, like git repack waiting a bit before deleting old packfiles? > > This all depends on how you are managing your server. It is likely that you > could create your own maintenance that handles this for you. > > The "git multi-pack-index (expire|repack)" cycle is built to prevent this sort > of issue, but is not yet integrated well with reachability bitmaps. You likely > require the bitmaps to keep your server performance, so that may not be a way > forward for you. We manage this on our servers with a repack wrapper that first creates hard links for all packfiles into a objects/pack/preserved dir and then we have patches on top of JGit [1] that actually know how to recover objects from that dir when the original pack is removed by repacking. It’s worked quite well for us for a couple years now and should be compatible with/without bitmaps (haven’t specifically tested) and any pack/repacking strategy. [1] https://git.eclipse.org/r/122288
