On 04/15/2011 05:48 PM, Zeev Tarantov wrote:
From: Zeev Tarantov<zeev.tarantov@xxxxxxxxx>
Google's Snappy data compression library is a faster alternative to LZO,
optimized for x86-64. On compressible input it compresses ~2.5x faster than LZO
and decompresses ~1.5-2x faster than LZO. On incompressible input, it skips the
input at 100x faster than LZO and decompresses ~4x faster than LZO.
It is released under BSD license.
This is a kernel port from user space C++ code.
The current intended use is with zram (see next patch in series).
Signed-off-by: Zeev Tarantov<zeev.tarantov@xxxxxxxxx>
These de/compression statistics are really good. The way I'm planning to
use snappy is to first determine compressibility of a page (unlike LZO,
it does not slow down incredibly on incompressible page) and then use
"heavier" compressors like zlib (which has ~5x overhead than LZO)
if snappy could not compress it well.
I'm in the process of writing a simple test for all these algorithms:
http://code.google.com/p/compcache/source/browse/sub-projects/fstats/fstats.c
This compresses input file page-by-page and dumps compressed sizes and
time taken to compress. With this data, I hope to come up with an
adaptive scheme which uses different compressors for different pages
such that overall compression ratio stays good while not hogging CPU
like when zlib is used alone.
As another use case, snappy can help with page de-duplication too which
zcache (or even zram) aims to support.
- Only the pages which compress to the same length have any chance of
being identical. This reduces the search space and allows more efficient
implementations (different tree for each size class).
- Compressed page is a smaller "value" to deal with. With appropriate
data structures, this may help make lookups faster.
In any case, I think we will have to keep LZO even if snappy
consistently outperforms it, both in terms of speed and compression
ratio, since zram is used on non-x86-64 hardware too (ARM) where snappy
is still untested.
Thanks for working on this!
Nitin
_______________________________________________
devel mailing list
devel@xxxxxxxxxxxxxxxxxxxxxx
http://driverdev.linuxdriverproject.org/mailman/listinfo/devel
