On Fri, Jul 26, 2013 at 09:20:34AM -0400, Jörn Engel wrote: > > I don't think the e2compr patches are strictly necessary. They are a > good option, but not the only one. Sorry for not chiming in earlier; I've been travelling this past week, and between that and a bunch of other things I've fallen a bit earlier on my e-mail. > One trick to simplify the problem is to make Dhaval's compressed files > strictly read-only. It will require some dance to load the compressed > content, flip the switch, then uncompress data on the fly and disallow > writes. Not the most pleasing of interfaces, but yet another option. Yeah, this is something that I've wanted for a while. (In fact a few years ago I shopped around this design to some folks who were associated with Firefox.) MacOS has something rather similar to this. I haven't had a chance to look at Dhaval's patches yet, but the way I've been thinking about this is that the compression and building the table mapping compressed clusters to byte offsets in the file would be done in userspace. Once the compressed file plus the table is written to the disk, the userspace program would then close the file descriptor, and then set the "compressed" bit. When the bit is set, we flush all of its pages from the page cache, and the file becomes immutable. At that point, the kernel will handle the decompression, by implementing readpages() by reading the pages into the buffer cache, and then decompressing the compressed cluster of pages into the page cache. This gives us transparent compression, with a fraction of the complexity of supporting read/write compression. In addition, since we don't have to worry rewriting a cluster (and having the modified compressed cluster taking up more space), the on-disk representation can be a lot more efficient, since you don't have to use a stacker-style design. One of the cool things about this design is that the vast majority of files on a typical distribution are write-once, and better yet, they are written by the package manager. So once you teach dpkg, rpm, and the Android package installer how to write the file in this compressed format and set the compressed bit, we can the vast majority of the benefits of using compressed file with minimal effort. - Ted P.S. This is interesting not just for systems with slow HDD's, but also for cheap, single-channel MMC flash, the kind found in low-end handset and embedded systems. P.P.S. At least in theory, nothing of what I've described here has to be ext4 specific. We could implement this in the VFS layer, at which point not only ext4 would benefit, but also btrfs, xfs, f2fs, etc. -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
