slow object packing during push

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have a large, funny repository that has very slow pushes unless I set pack.usebitmaps=false to false.

First, a description of the repo: it's about 175GB, and was created by combining about 40,000 smaller repositories.  Historically, these repos were submodules of one meta repository[2].  I have stitched together the submodules, and this is the repository in which the stitching was done - that is, it contains all of the objects from the smaller repos, plus all of the objects from the meta repository, plus the newly-created trees & commits for the stitched repositories.  As new commits come into the meta repository (which have gitlinks to new submodule commits), we fetch from the meta repository (8s - it would be 2s if we were fetching into a normal clone without all of the other stuff), and the submodules (up to 10s per and embarrassingly parallel). Then we stitch (~0s), and push to the stitched "unity" repository (~2 minutes!!!).  The entire repo fits in RAM (yes, all 175G) and is in fact in the disk cache (I prewarmed the cache before testing anything).  

The vast majority of the time appears to be spent in git pack-objects, and in particular in the stack trace in [1].  If I set pack.usebitmaps=false, the push only takes 10s.   This seems like pack bitmaps are a severe pessimization for my purposes.  This is true even immediately after a repack (that is, almost all of the objects are in one giant pack, except the newly-fetched ones).  I also tried setting up pack islands - one for each smaller repo, one for the stitched commits, and one for commits from the meta repo.  I'm not sure if this is necessary, but it's definitely not sufficient (my current config has it turned on, because I didn't feel like repacking again after testing it, and I tested it before testing pack.usebimaps). 


[1]
#9  0x000055d849183bfe in traverse_trees_and_blobs (ctx=ctx@entry=0x7fff2de42a80, 
    base=base@entry=0x7fff2de42a30) at list-objects.c:344
#10 0x000055d849183d2b in do_traverse (ctx=ctx@entry=0x7fff2de42a80) at list-objects.c:388
#11 0x000055d84918408f in traverse_commit_list_filtered (
    filter_options=filter_options@entry=0x55d849544e80 <filter_options>, 
    revs=revs@entry=0x7fff2de43f00, show_commit=show_commit@entry=0x55d8491a7830 <show_commit>, 
    show_object=show_object@entry=0x55d8491ac920 <show_object>, 
    show_data=show_data@entry=0x7fff2de42b50, omitted=omitted@entry=0x0) at list-objects.c:421
#12 0x000055d8491a8c1a in find_objects (bitmap_git=bitmap_git@entry=0x55d84a41cd40, 
    revs=revs@entry=0x7fff2de43f00, roots=0x0, seen=seen@entry=0x0, 
    filter=filter@entry=0x55d849544e80 <filter_options>) at pack-bitmap.c:603
#13 0x000055d8491af68d in prepare_bitmap_walk (revs=revs@entry=0x7fff2de43f00, 
    filter=filter@entry=0x55d849544e80 <filter_options>) at pack-bitmap.c:1004
#14 0x000055d8490b1983 in get_object_list_from_bitmap (revs=0x7fff2de43f00)
    at builtin/pack-objects.c:3294
#15 get_object_list (av=<optimized out>, ac=<optimized out>) at builtin/pack-objects.c:3373
#16 cmd_pack_objects (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>)
    at builtin/pack-objects.c:3739
#17 0x000055d84903ed19 in run_builtin (argv=<optimized out>, argc=<optimized out>, 
    p=<optimized out>) at git.c:450
#18 handle_builtin (argc=7, argv=0x7fff2de45320) at git.c:700
#19 0x000055d84903fd96 in run_argv (argv=0x7fff2de450a0, argcp=0x7fff2de450ac) at git.c:767
#20 cmd_main (argc=<optimized out>, argv=<optimized out>) at git.c:898
#21 0x000055d84903e8ef in main (argc=8, argv=0x7fff2de45318) at common-main.c:52

[2] https://github.com/twosigma/git-meta





[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux