Re: Poor push performance with large number of refs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Dec 09, 2014 at 09:41:28PM -0800, Shawn Pearce wrote:
> On Tue, Dec 9, 2014 at 4:37 PM, brian m. carlson
> <sandals@xxxxxxxxxxxxxxxxxxxx> wrote:
> > Most of the time is spent between the "Pushing to remote machine" and
> > "Counting objects", running git pack-objects:
> >
> >   git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
> >
> > Unfortunately, -vvv doesn't provide any helpful output.  I have some
> > suspicions what's going on here, but no hard data.  Where should I
> > be looking to determine the bottleneck?
> 
> My guess is the revision queue is struggling to insert 20,000 commits
> that the remote side "has", are uninteresting, and should not be
> transmitted. This queue insertion usually requires parsing the commit
> object out of the local object store to get the commit timestamp, then
> bubble sort inserting that commit into the queue.

I looked at this more in depth today and I found that the bottleneck is
--thin.  I tried git send-pack, which does not use --thin by default,
which led me to further testing.  A particular push went from 24 seconds
with --thin to 4 seconds without.

I agree that the large number of refs is at least part of the problem,
because reducing the number of refs has a slight but noticeable impact.
It's also the factor I can least control.

I have a patch which allows per-remote configuration of whether to use
thin packs (which I will send shortly), but I'm wondering if we can do
better, especially since --thin is the default.  It looks like --thin
forces pack-objects to do its own lookup (essentially a rev-list)
instead of using the values provided on stdin.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

Attachment: signature.asc
Description: Digital signature


[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]