Re: Forcing GC to always fail

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Nov 27, 2018 at 5:55 PM Elijah Newren <newren@xxxxxxxxx> wrote:
>
> On Tue, Nov 27, 2018 at 4:16 PM Ævar Arnfjörð Bjarmason
> <avarab@xxxxxxxxx> wrote:
> >
> > On Wed, Nov 28 2018, Bryan Turner wrote:
> >
> > > On Tue, Nov 27, 2018 at 3:47 PM Ævar Arnfjörð Bjarmason
> > > <avarab@xxxxxxxxx> wrote:
> > >>
> > >> On Tue, Nov 27 2018, Bryan Turner wrote:
> > >>
> > >> >
> > >> > Is there anything I can set, perhaps some invalid configuration
> > >> > option/value, that will make "git gc" (most important) and "git
> > >> > reflog" (ideal, but less important) fail when they're run in our
> > >> > repositories? Hopefully at that point customers will reach out to us
> > >> > for help _before_ they corrupt their repositories.
> > >>
> > >>     $ stahp='Bryan.Turner.will.hunt.you.down.if.you.manually.run.gc' && git -c gc.pruneExpire=$stahp gc; git -c gc.reflogExpire=$stahp reflog expire
> > >>     error: Invalid gc.pruneexpire: 'Bryan.Turner.will.hunt.you.down.if.you.manually.run.gc'
> > >>     fatal: unable to parse 'gc.pruneexpire' from command-line config
> > >>     error: 'Bryan.Turner.will.hunt.you.down.if.you.manually.run.gc' for 'gc.reflogexpire' is not a valid timestamp
> > >>     fatal: unable to parse 'gc.reflogexpire' from command-line config
> > >
> > > Thanks for that! It looks like that does block both "git gc" and "git
> > > reflog expire" without blocking "git pack-refs", "git repack" or "git
> > > prune". Fantastic! The fact that it shows the invalid value means it
> > > might also be possible to at least provide a useful hint that manual
> > > GC is not safe.
> > >
> > > I appreciate your help, Ævar.
> >
> > No problem. I was going to add that you can set
> > e.g. pack.windowMemory='some.message' to make this go for git-repack
> > too, but it sounds like you don't want that.
> >
> > Is there a reason for why BitBucket isn't using 'git-gc'? Some other
> > hosting providers use it, and if you don't run it with "expire now" or
> > similarly aggressive non-default values on an active repository it won't
> > corrupt anything.

We did use "git gc" (sans options; its configuration was applied via
"config") for several years, but there were some rough edges.

The biggest one was that "git gc" has a canned set of steps it runs
that can't be disabled, even when they add no value (or are actively
detrimental).

For us, the biggest issue was "git gc"'s insistence on trying to run
"git reflog expire". That triggers locking behaviors that resulted in
very frequent GC failures--and the only reflogs Bitbucket Server (by
default) creates are all configured to never ex[ire or be pruned, so
the effort is all wasted anyway.

Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 270 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: error: cannot lock ref 'stash-refs/pull-requests/13996/merge':
ref 'stash-refs/pull-requests/13996/merge' is at
2ec6a74b453d76f7a5247baa9f396361027ffdf but expected
1678f303202010c6d7e6201226df08dc8fc49ae3
remote: error: cannot lock ref 'stash-refs/pull-requests/13996/to':
ref 'stash-refs/pull-requests/13996/to' is at
bd32d53e4fe63b15be029085f1b6d795d526adbc but expected
f55ec06e89a11b8bdbcd97680a900361307d28c4
remote: error: cannot lock ref 'stash-refs/pull-requests/14006/merge':
ref 'stash-refs/pull-requests/14006/merge' is at
1cc907e2a7082033d70a164f222d3cce17a453a9 but expected
ae057003d7ed7d096b5b952191d784113f25b982
remote: error: cannot lock ref 'stash-refs/pull-requests/14006/to':
ref 'stash-refs/pull-requests/14006/to' is at
bd32d53e4fe63b15be029085f1b6d795d526adbc but expected
f55ec06e89a11b8bdbcd97680a900361307d28c4
remote: error: cannot lock ref 'stash-refs/pull-requests/14043/merge':
ref 'stash-refs/pull-requests/14043/merge' is at
a2e510b1b2b583b273f6d6d28e13151619e8d143 but expected
7735a5bde21815d307c68244e8fd2d67a09d5a39
remote: error: cannot lock ref 'stash-refs/pull-requests/14043/to':
ref 'stash-refs/pull-requests/14043/to' is at
bd32d53e4fe63b15be029085f1b6d795d526adbc but expected
f55ec06e89a11b8bdbcd97680a900361307d28c4
remote: error: cannot lock ref 'stash-refs/pull-requests/14047/merge':
ref 'stash-refs/pull-requests/14047/merge' is at
bd4f0e9bcbed34fd9befa65763f5aee6c9ebd8ce but expected
649dea948e8e6b54506615e5d61c6779c242d5af
remote: error: cannot lock ref 'stash-refs/pull-requests/14047/to':
ref 'stash-refs/pull-requests/14047/to' is at
bd32d53e4fe63b15be029085f1b6d795d526adbc but expected
f55ec06e89a11b8bdbcd97680a900361307d28c4
remote: error: failed to run reflog
To https://...
   f55ec06..bd32d53  master -> master

(Note: That example was from when "git gc --auto" was running attached
to a push, because auto-detach doesn't always detach, but our explicit
"git gc" processing would fail with the same "cannot lock ref"
messages.)

The worktree and rerere cleanup "git gc" does is also unnecessary
overhead, but was less of a concern because it wouldn't make GC
_fail_.

Another issue with the canned steps for "git gc" is that it means it
can't be used to do specific types of cleanup on a different schedule
from others. For example, we use "git pack-refs" directly to
frequently pack the refs in our repositories, separate from "git
repack" + "git prune" for repacking objects. That allows us to keep
our refs packed better without incurring the full overhead of
constantly building new packs.

It may be possible to make "git gc" run more targeted operations, but
even looking through the C code I couldn't find a way.

>
> ...assuming no other repo has this one as an alternate, which I
> suspect is the issue at play.  (I wrote an alternate-aware gc script
> years ago when using Atlassian Stash to try to workaround this issue,
> but think I only used it for a couple repos and never got around to
> deploying it in prod for continuous use, probably worried I had missed
> a corner case.  Had meant to, but at some point the powers that be
> decided to push us toward a different repository manager tool, and
> I've long since forgotten most details.)

"git gc" should be perfectly safe to run in a Bitbucket Server
repository which has forks; we "simply" configure it to never prune
(and that's applied in "config", so even if an administrator runs "git
gc" manually, _as long as they don't provide explicit arguments to the
contrary_, it still won't prune). When Git 2.10 or newer is installed
on the server we use "git repack --keep-unreachable" to prevent
writing unreachable objects as loose, and we skip running "git prune"
at all. For older versions of Git, we detect where the "objects/17"
heuristic would still want to run GC immediately after GC has already
run and we stream the loose object IDs to "git pack-objects" to write
a "garbage" pack, and then run "git prune-packed" to remove all the
loose objects.

It's on the backlog to actually start pruning again, even in forked
repositories, but there are some other design changes we need to make
first before we can ship that. (Not to mention the enormous amount of
testing we need to do to ensure it's safe...)

Bryan




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux