On Wed, May 29, 2019 at 9:27 PM Josh Boyer <jwboyer@xxxxxxxxx> wrote: > > On Wed, May 29, 2019 at 6:07 PM Neal Gompa <ngompa13@xxxxxxxxx> wrote: > > > > On Wed, May 29, 2019 at 5:53 PM Josh Boyer <jwboyer@xxxxxxxxxxxxxxxxx> wrote: > > > > > > On Wed, May 29, 2019 at 4:20 PM Ben Cotton <bcotton@xxxxxxxxxx> wrote: > > > > > > > > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > > > > > > > = Switch RPMs to zstd compression = > > > > > > > > == Summary == > > > > Binary RPMs are currently compressed with xz level 2. > > > > Switching to zstd would increase decompression speed significantly. > > > > > > > > == Owner == > > > > * Name: [[User:dmach| Daniel Mach]] > > > > * Email: dmach@xxxxxxxxxx > > > > > > > > == Detailed Description == > > > > * The change requires setting a new compression algorithm in rpm > > > > macros. Then a mass rebuild of all packages is required. > > > > > > The gcc team often does mass rebuilds on the side prior to updating > > > gcc in Fedora. Would it be possible to do the same or leverage their > > > rebuild work with the default changed in RPM to see what the true > > > overall savings is? That would get us a lot more data to see if it's > > > truly going to benefit the distro in terms of size and installation > > > speed. > > > > > > > This is news to me, as I've never heard of any "side mass rebuilds". > > They're prohibitively expensive to do, which is why we do only one per > > release anyway. > > They do them quite frequently outside of the Fedora infrastructure. > I'm not surprised. Fedora infrastructure tooling isn't great at allowing people to experiment with big changes. When I did the pkgconf change, I did a chunk of it in COPR and in a local OBS instance because it's basically impossible to test big changes in Fedora, which limits what kind of contributors can make big changes. It might just annoy me enough to talk about it at Flock, even... > > > > * The macro for setting the compression is: %define _binary_payload w19.zstdio > > > > * The recommended compression level is 19. The builds will take > > > > longer, but the additional compression time is negligible in the total > > > > build time and it pays off in better compression ratio than xz lvl2 > > > > has. > > > > * SRPM payload compression should stay at gzip (there's almost no > > > > benefit in changing the compression, because SRPM's contents is > > > > compressed already) > > > > > > > > === Use case: Firefox installation === > > > > I rebuilt firefox-66.0.5-1.fc30 with zstd level19. > > > > Then I compared installation times with the original (xz compressed) package: > > > > > > > > {| class="wikitable" > > > > |- > > > > ! Compression !! Target File System !! Time > > > > |- > > > > | xz level 2 || tmpfs || 8s > > > > |- > > > > | xz level 2 || ext4 on nvme || 11s > > > > |- > > > > | zstd level 19 || tmpfs || 2s > > > > |- > > > > | zstd level 19 || ext4 on nvme || 4s > > > > |- > > > > |} > > > > > > > > > > > > === Comparison of compression algorithms and levels === > > > > Following table shows '''cpio''' and '''compressed cpio''' extraction > > > > times into a tmpfs. Actual times in decompressing RPMs will differ due > > > > to extracting on an actual disk and also some overhead in the RPM tool > > > > (checks, scriptlets). > > > > > > > > {| class="wikitable" > > > > |- > > > > ! Compression !! Level !! Size B !! Size GiB > > > > !! Compression time !! Compression time, 4 threads !! > > > > Decompression time !! Comment > > > > |- > > > > | CPIO || - || 5016785692 || 4,7 > > > > || - || - || - > > > > || > > > > |- > > > > | xz || 2 || 1615017616 || 1,6 > > > > || 9m55s || - || 1m36s > > > > || slow decompression > > > > |- > > > > | pxz || 2 || 1631869880 || 1,6 > > > > || - || 6m11s || 1m38s > > > > || slow decompression > > > > |- > > > > | gzip || 9 || 2086354992 || 2,0 > > > > || 10m23s || - || 31s > > > > || insufficient compression ratio > > > > |- > > > > | bzip2 || 9 || 1889161565 || 1,8 > > > > || 8m || - || 2m50s > > > > || very slow decompression; compression ratio could be > > > > better > > > > |- > > > > | zstd || 3 || 1913536587 || 1,8 > > > > || 31s || 29s || 6,5s > > > > || > > > > |- > > > > | zstd || 10 || 1737928978 || 1,7 > > > > || 3m27s || 2m34s || 6,3s > > > > || > > > > |- > > > > | zstd || 15 || 1717303256 || 1,7 > > > > || 9m37s || 6m34s || 6,3s > > > > || identical compression speed to xz; fast decompression; > > > > slightly worse compression ratio than xz > > > > |- > > > > | zstd || 17 || 1635525492 || 1,6 > > > > || 16m16s || 11m20s || 6,7s > > > > || > > > > |- > > > > | zstd || 19 || 1575843696 || 1,5 > > > > || 24m2s || 18m55s || 7,7s > > > > || > > > > |- > > > > |} > > > > > > > > == Benefit to Fedora == > > > > * Faster installations/upgrades of user systems > > > > * Faster koji builds (installations in build roots) > > > > * Faster container builds > > > > * Lower bandwidth on mirrors if we choose the highest compression level > > > > > > > > I'm pretty sure this would break DeltaRPMs, since none of the drpm > > software has been updated to handle zstd compression. Neither drpm nor > > deltarpm handle it today. > > > > > > == Scope == > > > > * Proposal owners: submit a patch to redhat-rpm-config > > > > * Other developers: redhat-rpm-config maintainer: include the patch > > > > and make a new build > > > > * Release engineering: [https://pagure.io/releng/issue/8345 #8345] > > > > mass rebuild is needed > > > > > > > > == Upgrade/compatibility impact == > > > > * RPM in Fedora supports zstd compression already (from Fedora 28, > > > > rpm-4.14.0-0.rc2.5.fc28). No impact on Fedora users is expected. > > > > * Fedora <= 27 and some other distros will not be able to decompress > > > > zstd-compressed RPMs. > > > > > > If we did this, wouldn't it make it very difficult to use tools like > > > mock on RHEL / CentOS 7 to build for Fedora 3x? Or does RHEL 7 RPM > > > support zstd? > > > > > > > We're pretty much screwed here. Also, since RHEL 8's rpm package does > > not have zstd support compiled in, it too cannot handle the RPMs. > > > > Cf. https://git.centos.org/rpms/rpm/blob/c8/f/SPECS/rpm.spec#_17-18 > > Hm. That is highly unfortunate. I would hope an RFE would be filed > to at least add support in for RHEL 8 if this is approved. Otherwise > we're literally just shooting our distribution ecosystem in the foot > for the benefit of only Fedora. > When Igor and I had been looking at zstd back in the F28/F29 cycle, I had hoped that zstd support would have been inherited from Fedora, since it was forked from F28 (I had figured this much out from some sleuthing). Of course, that wound up not being the case. > > > Does MBS's concept of platform modules help us build a module across > > > the RPM zstd-support boundary? I think it does, but I honestly can't > > > remember for sure and I'm not aware of the details that go into MBS > > > performing the build. > > > > > > > Why would this help? MBS does nothing useful in this regard. It just > > calls Koji to make builds. When built for a specific platform, it'll > > use the definitions of that platform. And since the platform maps to > > the distro release, it's effectively the same as normal packages. > > Right. So it helps take a single module and build the same sources > for all distributions. You get binary artifacts that are a result of > that distribution's toolchain. That was the intent of my question. > > I agree it would not help the binary RPMs from the F31/F32 modules run > on a distribution that doesn't support zstd. However, because the > module could be built for such distributions we can still offer it > there. From an end user perspective, they get the application they > wanted. The number of people that want to cross-install RPMs/modules > is going to be proportionally small. > It affects the third-party ecosystem, especially those who might be using Fedora builders to build cross-distro packages. Surprisingly enough, it does happen. I know of a few major companies whose products' RPMs are built on Fedora to target everyone, so it'd be really bad if that didn't at least work on RHEL 8 and (open)SUSE. It's easy enough to fix for (open)SUSE, but RHEL is a more complicated matter since there's no community contribution process. -- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
