On 25/02/28 07:55AM, Theodore Ts'o wrote: > On Thu, Feb 27, 2025 at 08:53:56PM -0500, Ethan Carter Edwards wrote: > > Lately, I have been thinking a lot about the lack of APFS support on > > Linux. I was wondering what I could do about that. APFS support is not > > in-tree, but there is a proprietary module sold by Paragon software [0]. > > Obviously, this could not be used in-tree. However, there is also an > > open source driver that, from what I can tell, was once planned to be > > upstreamed [1] with associated filesystem progs [2]. I think I would > > base most of my work off of the existing FOSS tree. > > > > The biggest barrier I see currently is the driver's use of bufferheads. > > I realize that there has been a lot of work to move existing filesystem > > implementations to iomap/folios, and adding a filesystem that uses > > bufferheads would be antithetical to the purpose of that effort. > > Additionally, there is a lot of ifndefs/C preprocessor magic littered > > throughout the codebase that fixes functionality with various different > > versions of Linux. > > I don't see the use of bufferheads as a fundamental barrier to the > mainline kernel; certainly not for staging. First of all, there are a > huge number of file systems which still use buffer heads, including: > > adfs affs befs bfs ecryptfs efs exfat ext2 ext4 fat > freevxfs gfs2 hfs hfsplus hpfs isofs jfs minix nilfs2 > ntfs3 ocfs2 omfs pstore qnx4 qnx6 romfs sysv udf ufs > Good to know. I did not realized so many fs's stil used them. I will not let that stop me from submitting a the driver to staging. I definitely plan on moving them over as time permits. > There are many reasons to move to folios, including better > performance, and making it easier to make sure things are done > correctly if you can take advantage of iomap. > > For example, with ext4 we plan to work towards moving to use folios > and iomap for the data plane operations for buffered write (we already > use iomap for Direct I/O, FIEMAP support, etc.) and while we might > want to move away from buffer heads for metadata blocks, we would need > to change the jbd2 layer to use some simplified layer that looks an > awful lot like buffer heads before we could do that. We might try to > fork buffer heads, and strip out everything we don't need, and then > merge that with jbd2's journal_head structure, for example. But > that's a mid-term to long-term project, because using bufferheads > doesn't actually hurt anyone. (That being said, if anyone wants to > help out with the project of allowing us to move jbd2 away from buffer > heads, let me know --- patches are welcome.) > > In any case, cleaning up preprocessor magic and other thigns that were > needed because the code was designed for out of tree use would be > something that I'd encourage you to focus on first, and then try a > proposal to submit it to staging. Will do. > > Cheers, > > - Ted > > P.S. Something that you might want to consider using is fstests (AKA > xfstests), which is the gold standard for file system testing in > Linux. I have a test appliance VM of xfstests, which you can find > here[1]. I test x86 and arm64 kernels using Google Cloud, and on > local systems, using qemu/kvm. For qemu/kvm testing, this is being > used on Debian, Fedora, OpenSuSE, and MacOS. > > [1] https://github.com/tytso/xfstests-bld > > For kernel development on my Macbook Air M2, I can build arm64 kernels > using Debian running in a Parallels VM, and then to avoid the double > virtualization overhead, I run qemu on MacOS using the hvf > accelerator. It shouldn't be hard to make this work on your Ashai > Linux development system; see more on that below. > > For more details about this test infrastructure, including its use on > Google Cloud see the presentation here[2]. I am using gce-xfstests to > perform continuous integration testing by watching a git branch, and > when it gets updated, the test manager (running in an e2-micro VM) > automatically starts a 4 CPU VM to build the kernel, and then launches > multiple 2 CPU VM's to test multiple file system configurations in > parallel --- for example, I am currently running over two dozen fs > kernels testing ext4, xfs, btrfs, and f2fs on a Linux-next branch > every day). Running a smoke test costs pennies. A full-up test of a > dozen ext4 configuration (a dozen VM's, running for 2 hours of wall > clock time), costs under $2 at US retail prices. For APFS, if you > start with a single configuration, with many of the tests disable > because APFS won't many of the advanced file systems of ext4 and xfs, > I'm guessing it will cost you less than 25 cents per test run. > > [2] https://thunk.org/gce-xfstests > > Or of course you can use qemu-xfstests or kvm-xfstests using local > compute. I do love though being able to fire off a set of tests, then > suspend my laptop, knowing that I will receive e-mail with the test > results when they are ready. > > If you are interested in trying to use this on Asahi linux, I'd > certainly be happy help you with it. I suspect modulo some > instructures about which packages are needed, it shouldn't be that > hard to run a test appliance. Building new versions of the appliance > does require a Debian build chroot, which might be tricker to set up > on Asahi, but that's not necessary while you are getting started. > > In any case, I strongly encourage file system developers to use > xfstests earlier rather than later. See the last slide of [2] for my > opinion of "File system development without testing". :-) Good to know. I certainly agree with that last slide ;). Filesystems are far too important to develop without a test suite. Tests have certainly saved my butt in other projects, I have no doubt that it is the same for fs devel. Thank you for your detailed response. I was nervous that this idea would not be well-recieved since there is an existing out-of-tree driver, but since I am a believer in "upstream-first", I wanted to give this a try. Thanks, Ethan
