On 04/19/2011 08:01 PM, Zeev Tarantov wrote:
On Tue, Apr 19, 2011 at 14:31, Nitin Gupta<ngupta@xxxxxxxxxx> wrote:
On 04/15/2011 05:48 PM, Zeev Tarantov wrote:
From: Zeev Tarantov<zeev.tarantov@xxxxxxxxx>
Google's Snappy data compression library is a faster alternative to LZO,
optimized for x86-64. On compressible input it compresses ~2.5x faster
than LZO
and decompresses ~1.5-2x faster than LZO. On incompressible input, it
skips the
input at 100x faster than LZO and decompresses ~4x faster than LZO.
It is released under BSD license.
This is a kernel port from user space C++ code.
The current intended use is with zram (see next patch in series).
Signed-off-by: Zeev Tarantov<zeev.tarantov@xxxxxxxxx>
These de/compression statistics are really good. The way I'm planning to use
snappy is to first determine compressibility of a page (unlike LZO, it does
not slow down incredibly on incompressible page) and then use "heavier"
compressors like zlib (which has ~5x overhead than LZO)
if snappy could not compress it well.
I'm in the process of writing a simple test for all these algorithms:
http://code.google.com/p/compcache/source/browse/sub-projects/fstats/fstats.c
This compresses input file page-by-page and dumps compressed sizes and time
taken to compress. With this data, I hope to come up with an adaptive scheme
which uses different compressors for different pages such that overall
compression ratio stays good while not hogging CPU like when zlib is used
alone.
I have extended the block compressor tester I've written for Dan Magenheimer
(http://driverdev.linuxdriverproject.org/pipermail/devel/2011-April/015127.html)
to show this data.
It compresses a file one page at a time, computes a simple histogram
of compressed size, keeps elapsed cpu time and writes the compressed
blocks (and index) so the original file can be restored (to prevent
cheating, basically).
Code: https://github.com/zeevt/csnappy/blob/master/block_compressor.c
Results: https://github.com/zeevt/csnappy/blob/master/block_compressor_benchmark.txt
Because people don't click links, results inlined:
fstats now does all this and gnuplot does histogram :)
Anyways, I don't have any issues with links and don't have any problems
copy pasting your stuff here if needed for context. Still, when posting
your patches, it would be better to keep some of these performance
numbers in the patch description.
In any case, I think we will have to keep LZO even if snappy consistently
outperforms it, both in terms of speed and compression ratio, since zram is
used on non-x86-64 hardware too (ARM) where snappy is still untested.
We agree:
http://driverdev.linuxdriverproject.org/pipermail/devel/2011-April/015112.html
"It should not replace LZO where it is not faster. I have no benchmark
data from non-x86 hardware"
OK.
Qemu emulating ppc32 and armv5 shows strange profiles.
Please, can anyone benchmark the code on non-x86 hardware? In user space is ok.
Thanks for working on this!
What do you think of the submitted patch as -is?
csnappy as a compile time option is definitely welcome.
Manual runtime switching is not really needed -- instead of more knobs
for this, it would be better to have some simple scheme to automatically
switch between available compressors.
Thanks,
Nitin
_______________________________________________
devel mailing list
devel@xxxxxxxxxxxxxxxxxxxxxx
http://driverdev.linuxdriverproject.org/mailman/listinfo/devel
