On Thu, May 30, 2019 at 11:44:23AM -0700, Samuel Sieb wrote:
> On 5/30/19 7:54 AM, Daniel Mach wrote:
> >>I think before approving such changes, owners need to do mass
> >>rebuilds on their own and provide a graph of changes in size
> >>between original compression format and new one(s).
> >Doing that equals to the mass rebuild.
> >I'd rather do an analysis and if the numbers look sane, I'd prefer
> >doing a mass rebuild in a side tag so we can use the builds if the
> >results are sane. Hammering koji with so many scratch builds
> >doesn't sound right to me.
>
> Is it possible to just recompress the rpms instead of doing a full build?
+1, we can certainly get approximate statistics without rebuilding packages.
I did a somewhat unscientific test with 1GB of packages from a mock cache:
$ rm -rf zstd; mkdir zstd; for i in /var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/*rpm; do rpm2cpio $i | zstd >zstd/$(basename $i .rpm).cpio.zstd -19 -;done
$ rm -rf xz; mkdir xz; for i in /var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/*rpm; do rpm2cpio $i | xz >xz/$(basename $i .rpm).cpio.xz -2 -;done
$ rm -rf zstd20; mkdir zstd20; time for i in zstd/*; do zstdcat $i | zstd >zstd20/$(basename $i) --ultra -20 -;done
$ du -sh /var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/ /tmp/{xz,zstd,zstd20}
1019M /var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/
985M /tmp/xz
946M /tmp/zstd
930M /tmp/zstd20
$ time xzcat /tmp/xz/* >/dev/null
xzcat /tmp/xz/* > /dev/null 78.45s user 0.78s system 99% cpu 1:19.72 total
$ time zstdcat /tmp/zstd/* >/dev/null
zstdcat /tmp/zstd/* > /dev/null 9.19s user 0.44s system 98% cpu 9.751 total
$ time zstdcat /tmp/zstd20/* >/dev/null
zstdcat /tmp/zstd20/* > /dev/null 8.82s user 0.50s system 99% cpu 9.394 total
Notes:
- this is all single-threaded, since rpm doesn't do multithreading.
I saw some discussion that multithreading is easier with zstd because it's
reproducible with zstd, but not entirely with xz. Enabling multithreading
would be very beneficial for compression.
- $ rpm -q zstd xz
zstd-1.4.0-1.fc29.x86_64
xz-5.2.4-3.fc29.x86_64
- machine:
$ lscpu
Architecture: x86_64
CPU(s): 12
...
NUMA node(s): 2
Model name: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
/var/ is on spinning rust, /tmp is RAM.
- I'm not providing timings of compression, because I only did one run and
this might not be reproducible. Unscientific impression was that zstd
was quite a bit slower when compressing.
Zbyszek
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
