On 2025/1/17 09:14, Neal Gompa wrote:
On Thu, Jan 16, 2025 at 8:00 PM Gao Xiang <hsiangkao@xxxxxxxxxxxxxxxxx> wrote:
(resend this due to lack of subscription. and I just quote some reply to express my basic ideas on this stuff)
There's one thing the EROFS developers never talk about, and that's metadata compression. EROFS doesn't do it, but Squashfs does, and it has done since 2002.
Did you notice that https://erofs.docs.kernel.org/en/latest/faq.html ?
The tests are all done in way to flater EROFS and put Squashfs in the worst light. For example see https://erofs.docs.kernel.org/en/latest/comparsion/dedupe.html. They deliberately disable Squashfs metadata compression because EROFS doesn't support it, and use 128K blocks despite Squashfs supporting 1Mbyte blocks (look for the line [1] SquashFS uses –b 131072 by default, -noI will disable its metadata compression.).
It just shows that SquashFS doesn't support extent-based deduplication (because SquashFS on-disk indexes just record compressed sizes and cannot be randomly indexed), since EROFS doesn't support metadata compression so I benchmarked against SquashFS and explicitly point out the metadata compression is off on the SquashFS side to show the benefits of deduplication. Why it's called unbiased?
Don't make me wrong, if -b 1048576 (1m chunks) is used, then the chunk deduplication possibility is low.
But not anyone cares image size (especially for smartphone) just by using large extent sizes (like 1MiB) and real-time performance is important for them, decompress 1MiB to get 4KiB random data is unacceptable for them.
By doing this they deliberately reduce the compression and speed of Squashfs in their tests. This IMHO is biased and unethical. But that is how it is.
Also if I were a random filesystem author, I will never argue just due to lack of development resource.
Whether EROFS will use for Fedora LiveOS finally, honestly I don't care (although I wish it could be used more widely), I will focus on new features that end users really care and it's really an input. I only care EROFS can be totally usable on RHEL/CentOS since we have customers to use these OSes and I will serve for their detailed use cases.
Please note, EROFS was once proposed to resolve SquashFS unacceptable random runtime performance issues on the high-performance Android scenerios, and almost all smartphone vendors on the market use EROFS now. As for other scenarios, I would just call it as a plus.
If users think metadata compression is useful to minimize their images, and I will consider to add this in my spare time (Because it's never useful for my paid job too, they even don't care about multithreaded compression). Also I did not ask random vendors to contribute EROFS, just because Bytedance or other vendors need it and they develop new features for their use cases.
I certainly would appreciate features like metadata compression, BCJ
filters, and anything to improve image creation time when -Ededupe is
used. I also wanted to use zstd compression instead of lzma (since it
decompresses way faster), but the compression performance isn't quite
there yet.
Since it's not my paid job, I will try my best to work them out in
my spare time. But really, for our own use cases we don't care much
about them, it's also called "a labor of love".
Runtime speed is really valuable for us because typical media that
Linux live environments are running from are slower than even most
mobile device flash storage modules. At worst, we're talking about
actually booting from a DVD (which is still a thing people do in some
places), but we also need maximal space savings because of download
bandwidth and fitting on small media.
Anyway, EROFS was once specifically designed for Android devices
initially, I will try to make it work better for other useful cases,
questions, issues, and features are much helpful for me to improve
the filesystem itself.
Thanks,
Gao Xiang
--
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
