On 12/10/07, Nicolas Pitre <nico@xxxxxxx> wrote: > On Fri, 7 Dec 2007, Jon Smirl wrote: > > > Using this config: > > [pack] > > threads = 4 > > deltacachesize = 256M > > deltacachelimit = 0 > > > > And the 330MB gcc pack for input > > git repack -a -d -f --depth=250 --window=250 > > > > complete seconds RAM > > 10% 47 1GB > > 20% 29 1Gb > > 30% 24 1Gb > > 40% 18 1GB > > 50% 110 1.2GB > > 60% 85 1.4GB > > 70% 195 1.5GB > > 80% 186 2.5GB > > 90% 489 3.8GB > > 95% 800 4.8GB > > I killed it because it started swapping > > > > The mmaps are only about 400MB in this case. > > At the end the git process had 4.4GB of physical RAM allocated. > > > > Starting from a highly compressed pack greatly aggravates the problem. > > Starting with a 2GB pack of the same data my process size only grew to > > 3GB with 2GB of mmaps. > > You said having reproduced the issue, albeit not as severe, with the > Linux kernel repo. I did just that: > > # to get the default pack: > $ git repack -a -f -d > > # first measurement with a repack from a default pack > $ /usr/bin/time git repack -a -f --window=256 --depth=256 > 2572.17user 5.87system 22:46.80elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k > 15720inputs+356640outputs (71major+264376minor)pagefaults 0swaps > > # do it again to start from a highly packed pack > $ /usr/bin/time git repack -a -f --window=256 --depth=256 > 2573.53user 5.62system 22:45.60elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k > 29176inputs+356664outputs (210major+274887minor)pagefaults 0swaps > > This is with pack.threads=2 on a P4 with HT, and I'm using the machine > for other tasks as well, but all measured time is sensibly the same for > both cases. Virtual memory allocation never reached 700MB in both cases > either. > This is the mail about the kernel pack, the one you quoted is a gcc run. The kernel repo has the same problem but not nearly as bad. Starting from a default pack git repack -a -d -f --depth=1000 --window=1000 Uses 1GB of physical memory Now do the command again. git repack -a -d -f --depth=1000 --window=1000 Uses 1.3GB of physical memory I suspect the gcc repo has much longer revision chains than the kernel one since the kernel repo is only a few years old. The Mozilla repo contained revision chains with over 2,000 revisions. Longer revision chains result in longer delta chains. So what is allocating the extra memory? Either a function of the number of entries in the chain, or related to accessing the chain since a chain with more entries will need to be accessed more times. I have a 168MB kernel pack now after 15 minutes of four cores at 100%. Here's another observation, the gcc objects are larger. Kernel has 650K objects in 190MB, gcc has 870K objects in 330MB. Average gcc object is 30% larger. How should the average kernel developer interpret this? > > Nicolas > -- Jon Smirl jonsmirl@xxxxxxxxx - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
