Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote:
> Jonathan Dieter wrote:
> > My proposal would be to make zchunk the rpm compression format for
> > Fedora.
> 
> Given that:
> 1. zchunk is based on zstd, which is typically less efficient in terms of
>    compression ratio than xz, depending on settings
>    (see, e.g., https://github.com/inikep/lzbench), and
> 2. zchunk can by design only compress chunks individually and not benefit
>    from the space savings of a solid archive with a global dictionary,
> I fear that this is going to significantly increase the size of the RPMs, 
> which matters:
> * for the initial downloads,
> * for storage (e.g., keepcache=1, local mirrors, etc.), and
> * for the people not using deltas for whatever reason.
> 
> I think zchunk makes a lot of sense for the metadata, but I am not convinced 
> that it is the right choice for the RPMs themselves.

I suspect the first is true, but zchunk does actually allow for a
global (per-file) dictionary that can be used to compress each chunk. 
The difficulty is that the dictionary has to stay the same between file
versions, or the chunk checksums won't match.  There would have to be
some thought put into how to generate and store the dictionaries.

As for how much bigger a zchunked rpm will be compared to an xz rpm, at
the moment it's a bit hand-wavy.  Based on zchunked repodata work I've
done, I think we might be looking at a size that's slightly smaller
than a gzipped rpm.  I won't know for sure until I put together a
proof-of-concept, but I want to make sure that there aren't any gaping
holes in the proposal before I do that.

Jonathan
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux