Am 11.09.2009 16:47, schrieb Linus Torvalds: > > > On Fri, 11 Sep 2009, René Scharfe wrote: >> >> Using zlib directly avoids the overhead of a pipe and of buffering the >> output for blocked writes; surprisingly (to me), it isn't any faster. > > In fact, it should be slower. > > On SMP, you're quite likely better off using the pipe, and compressing on > another CPU. Of course, it's usually the case that the compression is _so_ > much slower than generating the tar-file (especially for the hot-cache > case) that it doesn't matter or the pipe overhead is even a slowdown. > > But especially if generating the tar-file has some delays in it > (cold-cache object lookup, whatever), the "compress in separate process" > is likely simply better, because you can compress while the other process > is looking up data for the tar. Yes, that makes sense and can be seen here (quad core, Fedora 11, best of five consecutive runs, Linux kernel repo): # git v1.6.5-rc0 $ time git archive --format=tar v2.6.31 | gzip -6 >/dev/null real 0m16.591s user 0m19.769s sys 0m0.474s # git v1.6.5-rc0 + patch $ time ../git/git archive --format=tar.gz -6 v2.6.31 >/dev/null real 0m20.390s user 0m20.299s sys 0m0.088s User time is quite similar, real time is lower when using a pipe. But what has bugged me since I added zip support is this result: # git v1.6.5-rc0 $ time git archive --format=zip -6 v2.6.31 >/dev/null real 0m16.471s user 0m16.340s sys 0m0.128s I'd have expected this to be the slowest case, because it's compressing all files separately, i.e. it needs to create and flush the compression context lots of times instead of only once as in the two cases above. And it's sequential and uses zlib, just like the tar.gz format. I suspect the convenience function gzwrite() adds this overhead. Oh, I just discovered pigz (http://zlib.net/pigz/), a parallel gzip: # git v1.6.5-rc0, pigz 2.1.5 $ time git archive --format=tar v2.6.31 | pigz -6 >/dev/null real 0m6.251s user 0m21.383s sys 0m0.547s So pipes win. :) Still need to investigate why zip is as (relatively) fast as it is, though. René -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html