Chris,
Thanks for your feedback and comments, it's very valuable to me.
In my previous message, I mentioned that CPU is underutilized during installation. I haven't investigated further why, but I suspect it's due to the inefficiency caused by the usage of the loop device and/or inefficiency in the rsync itself.
In fact, I have an optimization to file next weekend on my to do list.
> All of the Live installations use rsync.
And that's what I propose to change: to use unsquashfs instead of rsync, preliminary benchmarks show 8x improvement in decompressing speed on local media for XZ on local storage.
On my PC configuration, that will require approx. 52.98 MiB/s sequential read performance from local media and approx. 181.19MiB writing speed to the destination media.
That level performance is not common among today's USB drives or optical media -- the installation speed will not be capped due to the CPU limits, but otherwise limited by the sequential read speed of the installation media. It means, selecting an algorithm with better compression ratio should reduce the installation time from commonly used USB storage.
Yes, Zstd consumes 12.24x less CPU user time while unsquashfs, but let's consider the practical application.
Will Zstd decrease the installation time, given the constraints and optimization above -- that's what I plan to investigate in upcoming weekends.
My proposal focuses on reducing the installation media size, and recommends to use certain compression options. But, I think, the final decision is to be made by FESCO.
On Mon, Jan 20, 2020 at 12:42 AM Chris Murphy <lists@xxxxxxxxxxxxxxxxx> wrote:
On Sun, Jan 19, 2020 at 8:41 AM Bohdan Khomutskyi <bkhomuts@xxxxxxxxxx> wrote:
>
> Hello,
>
> Thanks everyone for posting feedback.
> More benchmarking results are available at https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS, including the 'plain' SquashFS filesystem.
> After performing the tests, I personally recommend to use xz compression with 1MiB block size, without bcj, on a 'plain' squash filesystem -- this will lead to a reduction of 142MiB on the ISO, compared to the stock Fedora 31 Workstation x86_64 image.
> Alternative compression options, such as Zstd, are also mentioned in the change proposal.
Thanks for all the tests.
While I see the meaningfully reduced CPU hit of xz compressed images,
the proposal leaves a lot of performance improvement on the table by
not also enabling zstd as an option in the compose process. The tests
show zstd results, but the proposal doesn't mention zstd at all.
In particular for Workstation ISO, the CPU hit isn't worth the size
savings for regular users, let alone the recurring hit for releng
composes and QA's automated installation tests. It's a lot of CPU burn
at both ends of the candle, for not a lot of size savings. I'm not
convinced it's worth the extra hit on the create side for Zstd level
22, compared to Zstd 15 or 17.
I admit I'm biased toward the two endpoints: create and consume, not
distribution ,i.e the mirror donors. Their storage and bandwidth
concerns were evaluated with the RPM change from xz to zstd. So I'm
mystified by the bias for image size.
Anyway, I approve of the change but disappointed if it really doesn't
let Fedora release engineering the ability to choose (possibly based
on image type - maybe there's some benefit to using xz for raw and
qcow2 images).
--
Chris Murphy
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
