On 6/4/19 4:00 AM, Jason L Tibbitts III wrote:
"PM" == Panu Matilainen <pmatilai@xxxxxxxxxx> writes:
PM> Note that rpm doesn't support parallel zstd compression, and while
PM> it does for xz, that's not even utilized in Fedora.
Doing parallel xz compression has a surprising cost in compression ratio
which gets worse as the thread count increases (because it just splits
the input into independent blocks and compresses them separately). I
did start on a feature to have it enabled but then abandoned that after
realizing that it didn't really work as I'd hoped.
Yup, I know. More than one people have been down that route :)
That said, I do wonder how difficult it would be to do parallel zstd
compression/decompression within RPM. If it were possible then that
might help to obviate some of the downsides.
No idea, except that last I looked, zstd seemed to be the only kid in
town who can do parallel decompression at all. The current zstd support
in rpm is basically just an initial code drop that implements the
barebones compress/decompress functionality. Besides parallel
operations, it'd probably be worth trying to teach it to use a
dictionary for example (the charts at https://github.com/facebook/zstd
are pretty impressive on that)
PM> To me the sweet spot between compression efficiency and speed seems
PM> closer to 10 than 19 - yes at a minor loss in space but huge speedup
PM> in both compress and decompress times.
One problem is that I don't think anyone wants to see any quantifiable
regression in overall package size. Spins still struggle to fit within
fixed media sizes as the package set grows ever larger.
Sure, everything's a compromise. Personally I find the fixed media
battle one that was long lost already and low in the overall priorities
but that's just me.
- Panu -
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
