Re: RFC/Pull Request: Refs db backend

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Jun 23, 2015 at 02:18:36PM -0400, David Turner wrote:

> > Can you describe a bit more about the reflog handling?
> > 
> > One of the problems we've had with large-ref repos is that the reflog
> > storage is quite inefficient. You can pack all the refs, but you may
> > still be stuck with a bunch of reflog files with one entry, wasting a
> > whole inode. Doing a "git repack" when you have a million of those has
> > horrible cold-cache performance. Basically anything that isn't
> > one-file-per-reflog would be a welcome change. :)
> 
> Reflogs are stored in the database as well.  There is one header entry
> per ref to indicate that a reflog is present, and then one database
> entry per reflog entry; the entries are stored consecutively and
> immediately following the header so that it's fast to iterate over them.

OK, that make sense. I did notice that the storage for the refdb grows
rapidly. If I add a millions refs (like refs/tags/$i) with a simple
reflog message "foo", I ended up with a 500MB database file.

That's _probably_ OK, because a million is getting into crazy
territory[1].  But it's 500 bytes per ref, each with one reflog entry.
Our ideal lower bound is probably something like 100 bytes per reflog
entry:

  - 20 bytes for old sha1
  - 20 bytes for new sha1
  - ~50 bytes for name, email, timestamp
  - ~6 bytes for refname (1000000 is the longest unique part)

That assumes we store binary[2] (and not just the raw reflog lines), and
reconstruct the reflog lines on the fly. It also assumes we use some
kind of trie-like storage (where we can amortize the cost of storing
"refs/tags/" across all of the entries).

Of course that neglects lmdb's overhead, and the storage of the ref tip
itself. But it would hopefully give us a ballpark for an optimal
solution. We don't have to hit that, of course, but it's food for
thought.

[1] The homebrew/homebrew repository on GitHub has almost half a million
    ref updates. Since this is storing not just refs but all ref
    updates, that's actually the interesting number (and optimizing the
    per-reflog-entry size is more interesting than the per-ref size).

[2] I'm hesitant to suggest binary formats in general, but given that
    this is a blob embedded inside lmdb, I think it's OK. If we were to
    pursue the log-structured idea I suggested earlier, I'm torn on
    whether it should be binary or not.

> > It has also been a dream of mine to stop tying the reflogs specifically
> > to the refs. I.e., have a spot for reflogs of branches that no longer
> > exist, which allows us to retain them for deleted branches.
> [...]
> That would be cool, and I don't think it would be hard to add to my
> current code; we could simply replace the header with a "tombstone".
> But I would prefer to wait until the series is merged; then we can build
> on top of it.

Yeah, I think you can add it easily to basically any system that does
not have the filesystem D/F conflicts in its storage (i.e., having
"refs/foo" does not block data under "refs/foo/bar").

> > But it may also be worth going with a slightly slower database if we can
> > get wider compatibility for free.
> 
> There's a JNI interface to LMDB, which is, of course, not native.  I
> don't think it would be too hard to entirely rewrite LMDB in Java, but
> I'm not going to have time to do it for the forseeable future.  I've
> asked Howard Chu if he knows of any efforts in progress.

Yeah, I think JNI is not enough for Eclipse folks. I don't think this is
a task that you would necessarily need to take on. More just something
to think about for the future when picking a format.

> Thanks, that's valuable.  For the refs backend, opening the LMDB
> database for writing is sufficient to block other writers.  Do you think
> it would be valuable to provide a git hold-ref-lock command that simply
> reads refs from stdin and keeps them locked until it reads EOF from
> stdin?  That would allow cross-backend ref locking.

I'm not sure what you would use it for. If you want to update the refs,
then you can specify a whole transaction with "git update-ref --stdin",
and that should work whatever backend you choose. Is there some other
operation you want where you hold the lock for a longer period of time?

-Peff
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]