On 21.01.2025 05:01, Theodore Ts'o wrote:
On Sun, Jan 19, 2025 at 03:37:27PM +0100, Gerhard Wiesinger wrote:
Are there any plans to include transparent compression with ext4 (especially
with zstd)?
I'm not aware of anyone in the ext4 deveopment commuity working on
something like this. Fully transparent compression is challenging,
since supporting random writes into a compressed file is tricky.
There are solutions (for example, the Stac patent which resulted in
Microsoft to pay $120 million dollars), but even ignoring the
intellectual property issues, they tend to compromise the efficiency
of the compression.
More to the point, given how cheap byte storage tends to be (dollars
per IOPS tend to be far more of a constraint than dollars per GB),
it's unclear what the business case would be for any company to fund
development work in this area, when the cost of a slightly large HDD
or SSD is going to be far cheaper than the necessary software
engineering investrment needed, even for a hyperscaler cloud company
(and even there, it's unclear that transparent compression is really
needed).
What is the business and/or technical problem which you are trying to
solve?
Regarding necessity:
We are talking in some scenarios about some factors of diskspace. E.g.
in my database scenario with PostgreSQL around 85% of disk space can be
saved (e.g. around factor 7).
In cloud usage scenarios you can easily reduce that amount of allocated
diskspace by around a factor 7 and reduce cost therefore.
You might also get a performance boost by using caching mechanism more
efficient (e.g. using less RAM).
Also with precompressed files (e.g. photo, videos) you can safe around
5-10% overall disk space which sounds less but in the area of several
hundred Gigabytes or even some Petabytes this is a lot of storage.
On evenly distributed data store you can save even more.
The technical topic is that IMHO no stable and practical usable Linux
filesystem which is included in the default kernel exists.
- ZFS works but is not included in the default kernel
- BTRFS has stability and repair issues (see mailing lists) and bugs
with compression (does not compress on the fly in some scenarios)
- bcachefs is experimental
Regarding patents: IMHO at least the STAC patents are all no longer
valid anymore.
Thnx.
Ciao,
Gerhard
