Re: Avoid race condition between fetch and repack/gc?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> On Mar 16, 2020, at 6:10 AM, Derrick Stolee <stolee@xxxxxxxxx> wrote:
> 
> On 3/16/2020 4:23 AM, Andreas Krey wrote:
>> Hi all,
>> 
>> we occasionally seeing things like this:
>> 
>> | DEBUG: 11:25:20: git -c advice.fetchShowForcedUpdates=false fetch --no-show-forced-updates -q --prune
> 
> I'm happy to see these options. I hope they are helping you!
> 
>> | Warning: Permanently added '[socgit.$company.com]:7999' (RSA) to the list of known hosts.
>> | remote: fatal: packfile ./objects/pack/pack-20256f2be3bd51b57e519a9f2a4d3df09f231952.pack cannot be accessed        
> This _could_ mean a lot of things, but....
> 
>> | error: git upload-pack: git-pack-objects died with error.
>> | fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
>> | remote: aborting due to possible repository corruption on the remote side.
>> | fatal: protocol error: bad pack header
>> 
>> and when you look in the server repository there is a new packfile dated just around
>> that time. It looks like the fetch tries to access a packfile that it assumes to exist,
>> but the GC on the server throws it away just in that moment, and thus upload-pack fails.
> 
> ...your intuition about repacking seems accurate. The important part of the
> race condition is likely that the server process read and holds a read handle
> on the .idx file, but when looking for the object contents it tries to open
> the .pack file which was deleted.
> 

[snip]

> 
>> Is there a way to avoid this?
>> 
>> Should there be, like git repack waiting a bit before deleting old packfiles?
> 
> This all depends on how you are managing your server. It is likely that you
> could create your own maintenance that handles this for you.
> 
> The "git multi-pack-index (expire|repack)" cycle is built to prevent this sort
> of issue, but is not yet integrated well with reachability bitmaps. You likely
> require the bitmaps to keep your server performance, so that may not be a way
> forward for you.

We manage this on our servers with a repack wrapper that first creates hard links for all packfiles into a objects/pack/preserved dir and then we have patches on top of JGit [1] that actually know how to recover objects from that dir when the original pack is removed by repacking. It’s worked quite well for us for a couple years now and should be compatible with/without bitmaps (haven’t specifically tested) and any pack/repacking strategy.

[1] https://git.eclipse.org/r/122288



[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux