I think I am getting some mixed signals from you guys, but I think
this is
what you meant.
I updated the first patches commit message to add a little more to the
conclusion, based on other e-mails in previous threads.
--
Mark A Rada (ferrous26)
marada@xxxxxxxxxxxx
-------->8-------------
From: Mark Rada <marada@xxxxxxxxxxxx>
Date: Thu, 30 Jul 2009 08:56:42 -0400
Subject: [PATCH 1/3] gitweb: add support for XZ compressed snapshots
The XZ compression format uses the LZMA2 compression algorithm, which
often yields higher compression ratios than both GZip and BZip2 at the
cost of using more CPU time and RAM. Though, while XZ is the slowest
for compression, it is much faster than BZip2 for decompression, almost
comparable to GZip (see benchmarks below).
You can enable XZ compressed snapshots by adding 'txz' to the list of
default options for snapshots in your $GITWEB_CONFIG.
I did some simple benchmarks, starting with an already tarballed
archive of the repos listed below. Memory usage seemed to be consistent
for any given algorithm at their default compression level. Timings were
gathered using the `time' command.
CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)
Virtual Memory Usage
GZip: 4152K BZip2: 13352K XZ: 102M
Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259) 349M
gzip 23.70s user 0.47s system 99% cpu 24.227 total 76M
gunzip 3.74s user 0.74s system 94% cpu 4.741 total
bzip2 130.96s user 0.53s system 99% cpu 2:11.97 total 59M
bunzip2 31.05s user 1.02s system 99% cpu 32.355 total
xz 448.78s user 0.91s system 99% cpu 7:31.28 total 51M
unxz 7.67s user 0.80s system 98% cpu 8.607 total
Git (0a53e9ddeaddad63ad106860237bbf53411d11a7) 11M
gzip 0.77s user 0.03s system 99% cpu 0.792 total 2.5M
gunzip 0.12s user 0.02s system 98% cpu 0.142 total
bzip2 3.42s user 0.02s system 99% cpu 3.454 total 2.1M
bunzip2 0.95s user 0.03s system 99% cpu 0.984 total
xz 12.88s user 0.14s system 98% cpu 13.239 total 1.9M
unxz 0.27s user 0.03s system 99% cpu 0.298 total
XZ (669413bb2db954bbfde3c4542fddbbab53891eb4) 1.8M
gzip 0.12s user 0.00s system 95% cpu 0.132 total 442K
gunzip 0.02s user 0.00s system 97% cpu 0.027 total
bzip2 1.28s user 0.01s system 99% cpu 1.298 total 363K
bunzip2 0.15s user 0.01s system 100% cpu 0.157 total
xz 1.62s user 0.03s system 99% cpu 1.652 total 347K
unxz 0.05s user 0.00s system 99% cpu 0.058 total
Purely from a time and memory perspective, nothing compares to GZip in
each of the three tests. Though, if you have an average upload speed of
20KB/s, it would take ~400 seconds longer to transfer the BZip2
compressed kernel snapshot than it would the XZ compressed snapshot,
the transfer time difference between GZip and XZ is even greater. The
wall clock time savings are relatively the same for all test cases, but
less dramatic for the smaller repositories.
It is also clear that XZ decompresses about 2x slower than GZip for
smaller snapshots only about 1.8x slower for the kernel snapshot. When
compared to BZip2, XZ is actually at least 2.7x faster, and gets
comparatively faster as the size of the snapshots increase (3.75x faster
than BZip2 for the kernel snapshot). However, this trend seems to
reverse itself for compression, it takes comparatively longer to
compress as snapshots get larger.
The obvious downside for XZ compressed snapshots is the large CPU and
memory load put on the server to actually generate a compressed
snapshot. Though XZ will eventually have good threading support, and I
suspect then that the wall clock time for making an XZ compressed
snapshot would go down considerably if the server had a beefy
multi-core CPU.
I have not enabled XZ compression by default because the current default
is GZip, and XZ is only really competitive with BZip2. XZ also requires
a lot of memory which could potentially lock up high load servers or
servers with little memory (until caching is available). Also, the XZ
format is still fairly new (the format was declared stable about 6
months ago), and there have been no "stable" releases of the utils yet.
Signed-off-by: Mark Rada <marada@xxxxxxxxxxxx>
---
gitweb/gitweb.perl | 8 ++++++++
1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 7fbd5ff..3398163 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -176,6 +176,13 @@ our %known_snapshot_formats = (
'format' => 'tar',
'compressor' => ['bzip2']},
+ 'txz' => {
+ 'display' => 'tar.xz',
+ 'type' => 'application/x-xz',
+ 'suffix' => '.tar.xz',
+ 'format' => 'tar',
+ 'compressor' => ['xz']},
+
'zip' => {
'display' => 'zip',
'type' => 'application/x-zip',
@@ -188,6 +195,7 @@ our %known_snapshot_formats = (
our %known_snapshot_format_aliases = (
'gzip' => 'tgz',
'bzip2' => 'tbz2',
+ 'xz' => 'txz',
# backward compatibility: legacy gitweb config support
'x-gzip' => undef, 'gz' => undef,
--
1.6.4
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html