On Wed, May 29, 2019 at 5:53 PM Josh Boyer <jwboyer@xxxxxxxxxxxxxxxxx> wrote: > > On Wed, May 29, 2019 at 4:20 PM Ben Cotton <bcotton@xxxxxxxxxx> wrote: > > > > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > > > = Switch RPMs to zstd compression = > > > > == Summary == > > Binary RPMs are currently compressed with xz level 2. > > Switching to zstd would increase decompression speed significantly. > > > > == Owner == > > * Name: [[User:dmach| Daniel Mach]] > > * Email: dmach@xxxxxxxxxx > > > > == Detailed Description == > > * The change requires setting a new compression algorithm in rpm > > macros. Then a mass rebuild of all packages is required. > > The gcc team often does mass rebuilds on the side prior to updating > gcc in Fedora. Would it be possible to do the same or leverage their > rebuild work with the default changed in RPM to see what the true > overall savings is? That would get us a lot more data to see if it's > truly going to benefit the distro in terms of size and installation > speed. > This is news to me, as I've never heard of any "side mass rebuilds". They're prohibitively expensive to do, which is why we do only one per release anyway. > > * The macro for setting the compression is: %define _binary_payload w19.zstdio > > * The recommended compression level is 19. The builds will take > > longer, but the additional compression time is negligible in the total > > build time and it pays off in better compression ratio than xz lvl2 > > has. > > * SRPM payload compression should stay at gzip (there's almost no > > benefit in changing the compression, because SRPM's contents is > > compressed already) > > > > === Use case: Firefox installation === > > I rebuilt firefox-66.0.5-1.fc30 with zstd level19. > > Then I compared installation times with the original (xz compressed) package: > > > > {| class="wikitable" > > |- > > ! Compression !! Target File System !! Time > > |- > > | xz level 2 || tmpfs || 8s > > |- > > | xz level 2 || ext4 on nvme || 11s > > |- > > | zstd level 19 || tmpfs || 2s > > |- > > | zstd level 19 || ext4 on nvme || 4s > > |- > > |} > > > > > > === Comparison of compression algorithms and levels === > > Following table shows '''cpio''' and '''compressed cpio''' extraction > > times into a tmpfs. Actual times in decompressing RPMs will differ due > > to extracting on an actual disk and also some overhead in the RPM tool > > (checks, scriptlets). > > > > {| class="wikitable" > > |- > > ! Compression !! Level !! Size B !! Size GiB > > !! Compression time !! Compression time, 4 threads !! > > Decompression time !! Comment > > |- > > | CPIO || - || 5016785692 || 4,7 > > || - || - || - > > || > > |- > > | xz || 2 || 1615017616 || 1,6 > > || 9m55s || - || 1m36s > > || slow decompression > > |- > > | pxz || 2 || 1631869880 || 1,6 > > || - || 6m11s || 1m38s > > || slow decompression > > |- > > | gzip || 9 || 2086354992 || 2,0 > > || 10m23s || - || 31s > > || insufficient compression ratio > > |- > > | bzip2 || 9 || 1889161565 || 1,8 > > || 8m || - || 2m50s > > || very slow decompression; compression ratio could be > > better > > |- > > | zstd || 3 || 1913536587 || 1,8 > > || 31s || 29s || 6,5s > > || > > |- > > | zstd || 10 || 1737928978 || 1,7 > > || 3m27s || 2m34s || 6,3s > > || > > |- > > | zstd || 15 || 1717303256 || 1,7 > > || 9m37s || 6m34s || 6,3s > > || identical compression speed to xz; fast decompression; > > slightly worse compression ratio than xz > > |- > > | zstd || 17 || 1635525492 || 1,6 > > || 16m16s || 11m20s || 6,7s > > || > > |- > > | zstd || 19 || 1575843696 || 1,5 > > || 24m2s || 18m55s || 7,7s > > || > > |- > > |} > > > > == Benefit to Fedora == > > * Faster installations/upgrades of user systems > > * Faster koji builds (installations in build roots) > > * Faster container builds > > * Lower bandwidth on mirrors if we choose the highest compression level > > I'm pretty sure this would break DeltaRPMs, since none of the drpm software has been updated to handle zstd compression. Neither drpm nor deltarpm handle it today. > > == Scope == > > * Proposal owners: submit a patch to redhat-rpm-config > > * Other developers: redhat-rpm-config maintainer: include the patch > > and make a new build > > * Release engineering: [https://pagure.io/releng/issue/8345 #8345] > > mass rebuild is needed > > > > == Upgrade/compatibility impact == > > * RPM in Fedora supports zstd compression already (from Fedora 28, > > rpm-4.14.0-0.rc2.5.fc28). No impact on Fedora users is expected. > > * Fedora <= 27 and some other distros will not be able to decompress > > zstd-compressed RPMs. > > If we did this, wouldn't it make it very difficult to use tools like > mock on RHEL / CentOS 7 to build for Fedora 3x? Or does RHEL 7 RPM > support zstd? > We're pretty much screwed here. Also, since RHEL 8's rpm package does not have zstd support compiled in, it too cannot handle the RPMs. Cf. https://git.centos.org/rpms/rpm/blob/c8/f/SPECS/rpm.spec#_17-18 > Does MBS's concept of platform modules help us build a module across > the RPM zstd-support boundary? I think it does, but I honestly can't > remember for sure and I'm not aware of the details that go into MBS > performing the build. > Why would this help? MBS does nothing useful in this regard. It just calls Koji to make builds. When built for a specific platform, it'll use the definitions of that platform. And since the platform maps to the distro release, it's effectively the same as normal packages. > > == How To Test == > > * dnf install <package> > > * rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package> > > * expected output: zstd 19 > > > > Also the overall system installation time should decrease significantly. > > > > == User Experience == > > See '''Benefit to Fedora''' > > > > == Dependencies == > > N/A > > > > == Contingency Plan == > > * Contingency mechanism: Not needed, Fedora will stay at current compression. > > * Contingency deadline: N/A > > This seems wrong. If we get through a mass rebuild (or partial mass > rebuild) and find some ugly unknown issue with zstd compression, we're > going to have to do another mass rebuild to revert everything back, > correct? That should be listed as the Contingency, even if it's > unlikely. > Yeah, if this turns out bad, we'd need a second mass build to eliminate packages with zstd compression. > > * Blocks release? No > > Are we not advocating for a fully successful mass rebuild? Would we > ship the distribution with only a portion (significant or otherwise) > switched to zstd? > I'd hope not... -- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
