On Wed, Sep 14, 2016 at 4:58 PM, Jeff King <peff@xxxxxxxx> wrote: > There's a fancy new compression algorithm called "zstd". The > idea is that it's supposed to get similar compression ratios > to zlib, but with much faster compression and decompression > times. And on top of that, a nice sliding scale to trade off > size versus time on the compression side. > > The zstd site at https://facebook.github.io/zstd/ claims > close to 3x speedup for both compression and decompression > versus zlib, with similar compression ratios. There are > other fast algorithms (like lz4), but they usually compress > much worse (follow the link above for a nice table of > results). > > Since any git operations that have to access objects need to > do a zlib inflate, in theory we can speed up everything by > using zstd. And then on the packing side, use higher > compression levels when making on-disk packfiles (which will > be accessed many times) and lower ones when making loose > objects, or deflating packed objects on the fly when serving > fetches. > > The catch, of course, is that it's a new incompatible > format. This would be a pretty huge change and totally break > backwards compatibility for git, not just on disk but > on-the-wire as well. So my goal here was not a finished > product but just a quick experiment to see if it did indeed > bring the promise speedups. > > Disappointingly, the answer seems to be "no". After having looked at the data, I disagree with the conclusion. And for that I think we need to reason about the frequency of the operations happening. * As an enduser, happily hacking away at one repository, I probably do not care about the pack size on disk as much as I care about timing of the local operations. And I assume that for each repack we have about 1000 reads (log/rev-list) The 1000 is a wild speculation without any data to back it up. So as an end user I'd be happy about [zstd, ~5] For the end user LZ4 seems to be the best solution if it were available. * As a service provider, I know we have a lot more reads than writes, and repacking is annoying. Also at that scale the disk isn't negligible cheap. So we need to weigh the numbers differently, but how? I suspect depending on the weighting it could still be considered beneficial to go with zstd5. (No hard numbers here)