On 6/4/19 8:31 AM, Panu Matilainen wrote:
On 6/4/19 4:00 AM, Jason L Tibbitts III wrote:
"PM" == Panu Matilainen <pmatilai@xxxxxxxxxx> writes:
PM> Note that rpm doesn't support parallel zstd compression, and while
PM> it does for xz, that's not even utilized in Fedora.
Doing parallel xz compression has a surprising cost in compression ratio
which gets worse as the thread count increases (because it just splits
the input into independent blocks and compresses them separately). I
did start on a feature to have it enabled but then abandoned that after
realizing that it didn't really work as I'd hoped.
Yup, I know. More than one people have been down that route :)
That said, I do wonder how difficult it would be to do parallel zstd
compression/decompression within RPM. If it were possible then that
might help to obviate some of the downsides.
No idea, except that last I looked, zstd seemed to be the only kid in
town who can do parallel decompression at all. The current zstd support
in rpm is basically just an initial code drop that implements the
barebones compress/decompress functionality. Besides parallel
operations, it'd probably be worth trying to teach it to use a
dictionary for example (the charts at https://github.com/facebook/zstd
are pretty impressive on that)
...except that of course rpm compression doesn't occur at individual
file level but the whole, prebuilt dictionaries aren't that useful with
the payload.
But there are other tuning options that seem more beneficial to the rpm
use-case, such as the zstd cli --long equivalent: in my testcase
compressing with --long at level 14 gives a slightly better compression
rate than level 19 without it, at a fraction of the time (40s vs
2min31s). It uses more memory (there never was a free lunch was there?)
but peanuts to what large compiles can use. At any rate, that is
something to look into.
- Panu -
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
