On Sun, 2018-11-18 at 13:19 -0500, Stephen John Smoogen wrote: > On Sun, 18 Nov 2018 at 12:49, Neal Gompa <ngompa13@xxxxxxxxx> wrote: > > On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter <jdieter@xxxxxxxxx > > > wrote: > > > On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > > > > Jonathan Dieter wrote: > > > > > My proposal would be to make zchunk the rpm compression > > > > > format for > > > > > Fedora. > > > > > > > > Given that: > > > > 1. zchunk is based on zstd, which is typically less efficient > > > > in terms of > > > > compression ratio than xz, depending on settings > > > > (see, e.g., https://github.com/inikep/lzbench), and > > > > 2. zchunk can by design only compress chunks individually and > > > > not benefit > > > > from the space savings of a solid archive with a global > > > > dictionary, > > > > I fear that this is going to significantly increase the size of > > > > the RPMs, > > > > which matters: > > > > * for the initial downloads, > > > > * for storage (e.g., keepcache=1, local mirrors, etc.), and > > > > * for the people not using deltas for whatever reason. > > > > > > > > I think zchunk makes a lot of sense for the metadata, but I am > > > > not convinced > > > > that it is the right choice for the RPMs themselves. > > > > > > I suspect the first is true, but zchunk does actually allow for a > > > global (per-file) dictionary that can be used to compress each > > > chunk. > > > The difficulty is that the dictionary has to stay the same > > > between file > > > versions, or the chunk checksums won't match. There would have > > > to be > > > some thought put into how to generate and store the dictionaries. > > > > > > As for how much bigger a zchunked rpm will be compared to an xz > > > rpm, at > > > the moment it's a bit hand-wavy. Based on zchunked repodata work > > > I've > > > done, I think we might be looking at a size that's slightly > > > smaller > > > than a gzipped rpm. I won't know for sure until I put together a > > > proof-of-concept, but I want to make sure that there aren't any > > > gaping > > > holes in the proposal before I do that. > > > > > > > I did some work several months ago to evaluate zstd compression for > > RPMs for Fedora, because of the lower memory and CPU usage for > > (de)compression. However, the average size increase from xz was > > pretty > > large (~20% or more on average, and nothing ever was either the > > same > > or smaller), even with heavier compression settings. That might > > have > > changed a bit with newer zstd releases that offer some more > > tunables, > > but I think it'll remain a tough sell on disk space. > > So there are at least 4 legs here: > CPU usage (in both uncompression install and deltarpm) > Memory usage per transaction > Network amount > Disk amount > > I expect that the best we are going to get in any 'improvement' is > going to be 3 out of the 4. The xz compression and delta-rpm has a > cpu/memory tradeoff for disk and network in comparison to gzip but it > is mostly acceptable if you have fairly modern desktops. However for > older hardware or lower power systems that tradeoff may not be good. Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't require extra CPU usage on the client side as they don't go through the decompress-recompress cycle that deltarpms do. Re-assembling a zchunk file requires no compression or decompression. On the client: The zchunk advantage over regular rpm is decreased network usage, while its disadvantage is increased disk usage (since the local chunks will be uncompressed). The zchunk advantage over deltarpms is much less CPU usage, while its disadvantages are slightly larger network usage and increased disk usage. Note that for most users the increased disk usage is temporary, since rpms are deleted after install. In our infrastructure: The zchunk advantage over deltarpms is that they are created in the build stage and shouldn't take any longer than building a normal rpm, while deltarpms take quite a while to build. The disadvantage is that our current rpms use xz compression which is more efficient at compressing a whole file than zchunk is, so zchunked rpms will require more disk space. Hope that helps clarify things. Jonathan _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
