Re: Git is not scalable with too many refs/*

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Just thought that I should add some numbers to this thread as it seems that
the later versions of git are worse off by several orders of magnitude on
this one.  

We have a Gerrit repo with just under 100K refs in refs/changes/*.  When I
fetch them all with git 1.7.6 it does not seem to complete.  Even after 5
days, it is just under half way through the ref #s!   It appears, but I am
not sure, that it is getting slower with time also, so it may not even
complete after 10 days, I couldn't wait any longer.  However, the same
command works in under 8 mins with git 1.7.3.3 on the same machine!

Syncing 100K refs:

  git 1.7.6      > 8 days?
  git 1.7.3.3   ~8mins

That is quite a difference!  Have there been any obvious changes to git that
should cause this?  If needed, I can bisect git to find out where things go
sour, but I thought that perhaps there would be someone who already
understands the problem and why older versions aren't nearly as bas as
recent ones.

Some more things that I have tried:  after syncing the repo locally with all
100K refs under refs/changes, I cloned it locally again and tried fetching
locally with both git 1.7.6 and 1.7.3.3.  I got the same results as
remotely, so it does not appear to be related to round trips.

The original git remote syncing takes just a bit of time, and then it
outputs lines like these:
 ...
 * [new branch]      refs/changes/13/66713/2 -> refs/changes/13/66713/2
 * [new branch]      refs/changes/13/66713/3 -> refs/changes/13/66713/3
 * [new branch]      refs/changes/13/66713/4 -> refs/changes/13/66713/4
 * [new branch]      refs/changes/13/66713/5 -> refs/changes/13/66713/5
 ...

This is the part that takes forever.  The lines seem to scroll by slower and
slower (with git 1.7.6).  In the beginning, the lines might be a screens
worth a minute, after 5 days, about 1  a minute.  My CPU is pegged at 100%
during this time (one core).  Since I have some good test data for this, let
me know if I should test anything specific.

Thanks,

-Martin

Employee of Qualcomm Innovation Center, Inc. which is a member of Code
Aurora Forum


--
View this message in context: http://git.661346.n2.nabble.com/Git-is-not-scalable-with-too-many-refs-tp6456443p6773496.html
Sent from the git mailing list archive at Nabble.com.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]