Re: Any tips for improving the performance of cloning large repositories?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 16 December 2011 14:14, Seth Robertson <in-gitvger@xxxxxxxx> wrote:
>
> In message <hbf.20111216zcin@xxxxxxxxxxxxx>, Hallvard Breien Furuseth writes:
>
>    I wrote:
>    > Do you often need to clone from a remote?  Instead of cloning from a
>    > local (git clone --mirror) which gets auto-updated from the remote.
>
>    Er, obviously not, since you tried that with rsync.  Create the mirror
>    with 'git clone --mirror', then update it with 'git fetch' rather than
>    rsync.
>
> If you really need to perform a full clone from the buildbot with or
> without a different working directory (for instance if you have
> buildbots/checkout users running in parallel where multiple users need
> a consistent HEAD for multiple sequential operations) then instead
> consider cloning with --reference or --shared.

Well that's counter intuitive....

 - reverting the original repo to one big pack speeds up the clone
 - adding a --local --reference mirror slows it down

Timings:

14:41 ajb@vsbldhost/i686 [ajb] >time git clone git://engbot/repo.git
test-clone-bigpack.git
Initialized empty Git repository in /scratch/ajb/test-clone-bigpack.git/.git/
remote: Counting objects: 371220, done.
remote: Compressing objects: 100% (88900/88900), done.
remote: Total 371220 (delta 274586), reused 371220 (delta 274586)
Receiving objects: 100% (371220/371220), 1.78 GiB | 20.10 MiB/s, done.
Resolving deltas: 100% (274586/274586), done.
Checking out files: 100% (42909/42909), done.

real    8m53.008s
user    2m53.151s
sys     7m16.339s

14:53 ajb@vsbldhost/i686 [ajb] >time git clone --local --reference
/var/cache/repos/repo.git git://engbot/repo.git te
st-clone-local.git
Initialized empty Git repository in /scratch/ajb/test-clone-local.git/.git/
Checking out files: 100% (42909/42909), done.

real    14m6.333s
user    1m6.844s
sys     12m44.676s

Two things are odd. The first is the clone "hung" at around 22%
checking out the files for ~ 10 minutes before finishing the remaining
70% in a few seconds. Secondly is seems in both cases the systime is
quite high.

-- 
Alex, homepage: http://www.bennee.com/~alex/
http://www.half-llama.co.uk
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]