On Mon, Jun 3, 2019 at 7:01 PM Jason L Tibbitts III <tibbs@xxxxxxxxxxx> wrote: > > >>>>> "PM" == Panu Matilainen <pmatilai@xxxxxxxxxx> writes: > > PM> Note that rpm doesn't support parallel zstd compression, and while > PM> it does for xz, that's not even utilized in Fedora. > > Doing parallel xz compression has a surprising cost in compression ratio > which gets worse as the thread count increases (because it just splits > the input into independent blocks and compresses them separately). I > did start on a feature to have it enabled but then abandoned that after > realizing that it didn't really work as I'd hoped. Which is also why parallel xz compression doesn't produce reproducible results. > That said, I do wonder how difficult it would be to do parallel zstd > compression/decompression within RPM. If it were possible then that > might help to obviate some of the downsides. At least for small files, and there are many in any distribution, using a dictionary very well could improve compression/decompression time, compression ratio, more than threads. Adding dictionary support would help all the single thread hardware, and even the builders when zstd -T0 option dictates there's only 1 or 2 threads available. On the generic sample set, it's functionally like getting 4 threads on speed, and even compression ratio goes up by ~3x. But I have no idea how that sample set compares to Fedora's files. -- Chris Murphy _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
