Greetings,
I am running “git-repack -A -d -f -F --window=250 --depth=250” on a Git repository converted using git-svn.
The repository contains more than 10 years of development, and a mixture of source code and media assets. The size of the objects directory is around 50GB, and there are around 700k objects. There are several svn branches.
I have ran the repack with Git 2.17.1 (original package from Ubuntu 18.04), 2.19.1 (package from Ubuntu Git PPA) and with git next (bc1bbc6f855c3b5ef7fcbd0f688f647c4e5b208b).
The system is a 16 core / 32 thread Threadripper with 128GB of RAM and NVMe storage. The repack starts strong, with 32 threads but it fairly quickly gets to 99% done and the number of threads drops to 4 then 3 then 2. However, running “dstat 5” I see lots of “sys” time without any IO time (the network traffic you see is caused by SSH).
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read writ| recv send| in out | int csw
24 24 48 4 0| 0 0 | 362B 8392B| 0 0 | 64k 5042
27 25 44 4 0| 0 0 | 289B 4830B| 0 0 | 36k 5017
24 24 48 4 0| 0 0 | 362B 8392B| 0 0 | 64k 5042
27 25 44 4 0| 0 0 | 289B 4830B| 0 0 | 36k 5017
20 22 54 4 0| 0 0 | 330B 4826B| 0 0 | 31k 4657
22 23 51 3 0| 0 0 | 286B 4834B| 0 0 | 27k 4536
17 23 56 4 0| 0 0 | 299B 4820B| 0 0 | 29k 4099
24 23 49 4 0| 0 0 | 347B 4830B| 0 0 | 32k 4436
27 24 45 4 0| 0 0 | 292B 4820B| 0 0 | 22k 4513
26 27 44 3 0| 0 0 | 292B 4820B| 0 0 | 24k 4878
25 28 44 4 0| 0 0 | 329B 4837B| 0 0 | 45k 5023
23 30 44 3 0| 0 9830B| 370B 4836B| 0 0 | 40k 5064
23 26 47 4 0| 0 0 | 373B 4848B| 0 0 | 52k 4746
25 26 45 4 0| 0 0 | 323B 4833B| 0 0 | 31k 4874
24 28 45 3 0| 0 0 | 269B 4842B| 0 0 | 30k 4820
25 28 43 4 0| 0 0 | 280B 4826B| 0 0 | 45k 5237
25 29 43 3 0| 0 0 | 263B 4826B| 0 0 | 26k 5037
22 30 45 4 0| 0 0 | 287B 4823B| 0 0 | 45k 4154
0 53 42 5 0| 0 0 | 275B 1210B| 0 0 |5017 1331
0 53 42 5 0| 0 0 | 97B 0 | 0 0 |5006 1272
0 53 42 5 0| 0 0 | 70B 0 | 0 0 |5008 1272
0 53 42 5 0| 0 0 | 24B 0 | 0 0 |5007 1283
11 43 43 4 0| 0 0 | 644B 19k| 0 0 | 20k 2995
25 28 43 4 0| 0 123k| 304B 4802B| 0 0 | 29k 5118
25 27 44 4 0| 0 0 | 297B 4800B| 0 0 | 41k 5121
26 27 43 4 0| 0 0 | 320B 4786B| 0 0 | 29k 4973
22 30 44 3 0| 0 0 | 332B 4786B| 0 0 | 32k 4610
24 27 46 3 0| 0 0 | 296B 4786B| 0 0 | 56k 4632
35 17 44 3 0| 0 0 | 292B 4776B| 0 0 | 100k 4889
34 20 43 4 0| 0 9830B| 610B 4820B| 0 0 | 59k 4811
This when it gets to two threads phase:
1 40 59 0 0| 0 38k| 373B 2458B| 0 0 | 50k 2874
0 40 59 0 0| 0 0 | 268B 2892B| 0 0 | 10k 2858
0 40 59 0 0| 0 0 | 299B 3102B| 0 0 |8392 2919
1 40 59 0 0| 0 0 | 297B 3176B| 0 0 | 21k 2831
0 40 59 0 0| 0 0 | 281B 3414B| 0 0 |9192 2856
1 40 59 0 0| 0 0 | 264B 3493B| 0 0 | 17k 2905
1 40 59 0 0| 0 35k| 340B 3571B| 0 0 | 13k 2885
The repack still progresses, as I see the count of repacked objects being incremented every several minutes. But I suspect there is some terrible lock contention, because otherwise I can’t explain the high system usage with no IO.
Running a strace on the running git-repack process shows only these:
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
Any idea on how to debug this? I have ran git-repack under gdb, but it seems to spin on builtin/repack.c line 409.
Thank you for Git,
florin
