This patch series implements a memory limit on the window size for pack-objects and repack. Basically, the window size will temporarily grow smaller than the --window option specifies if the total memory usage of the window is over the specified limit. With this series I can run even with --window=1000 on my troublesome repository (described in an email a couple of days ago; basically, it has a 20MB file with around 200 revisions, plus a bunch of normal sized files) with memory usage limited to 512MB and it will happily scale down the window to accommodate the file without blowing out my machine. The --window option still specifies the size of the window and is still required; the window will not grow to reach the memory limit, it will only shrink to fit. I think this may be a feature, as running with a very large window depth and a memory limit basically means that you will pack at an approximately constant slow speed, rather than rushing ahead as it does now for very small objects. I took the easy way out and expire objects out of the window after allocation has occurred, rather than figuring out how much needed to be cleared before allocating. This made the logic much more feasible, though. I chose --window-bytes=N and pack.windowBytes=N as my option and configuration names. I'm not in love with them, though; no other configuration in Git mentions bytes in the name as far as I can see, but I wanted to clearly differentiate it from the window "size" that --window gets you. The first patch in this series is optional, but I recommend it. It makes it so that files that are much smaller than a potential delta source don't even try to delta with it. This is handy for running in a mixed file size repository with a large window, as it means that when you get to small files again you start moving fast without having to wait for the large objects to trickle out of the window. The cut-off is currently if it is 1/32 the size, but that number was completely arbitrary. [PATCH 1/5] Don't try to delta if target is much smaller than source [PATCH 2/5] Support fetching the memory usage of a delta index [PATCH 3/5] Add pack-objects window memory usage limit [PATCH 4/5] Add --window-bytes option to git-repack [PATCH 5/5] Add documentation for --window-bytes, pack.windowBytes Documentation/config.txt | 5 +++ Documentation/git-pack-objects.txt | 8 +++++ Documentation/git-repack.txt | 8 +++++ builtin-pack-objects.c | 58 +++++++++++++++++++++++++++++++---- delta.h | 7 ++++ diff-delta.c | 10 ++++++ git-repack.sh | 3 +- 7 files changed, 91 insertions(+), 8 deletions(-) -bcd - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
