On Wed, Aug 30, 2023 at 04:07:39PM +0200, Christoph Hellwig wrote: > Hi all, > > we have a lot of on-disk file system drivers in Linux, which I consider > a good thing as it allows a lot of interoperability. At the same time > maintaining them is a burden, and there is a lot expectation on how > they are maintained. > > Part 1: untrusted file systems > > There has been a lot of syzbot fuzzing using generated file system > images, which I again consider a very good thing as syzbot is good > a finding bugs. Unfortunately it also finds a lot of bugs that no > one is interested in fixing. The reason for that is that file system > maintainers only consider a tiny subset of the file system drivers, > and for some of them a subset of the format options to be trusted vs > untrusted input. It thus is not just a waste of time for syzbot itself, > but even more so for the maintainers to report fuzzing bugs in other > implementations. > > What can we do to only mark certain file systems (and format options) > as trusted on untrusted input and remove a lot of the current tension > and make everyone work more efficiently? Note that this isn't even > getting into really trusted on-disk formats, which is a security > discussion on it's own, but just into formats where the maintainers > are interested in dealing with fuzzed images. I think this completely misses the point of contention of the larger syzbot vs filesystem discussion: the assertion that "testing via syzbot means the subsystem is secure" where "secure" means "can be used safely for operations that involve trust model violations". Fundamentally, syzbot does nothing to actually validate the filesystem is "secure". Fuzzing can only find existing bugs by simulating an attacker, but it does nothing to address the underlying issues that allow that attack channel to exist. All "syzbot doesn't find bugs" means is that -random bit manipulation- of the filesystem's metadata *hasn't found issues*. Even though the XFS V5 format is pretty robust against random bit manipulation, it's certainly not invulnerable and cannot detect coordinated, multiple object corruptions (cross linked blocks, cycles in trees, etc) without a full filesystem scan. These sorts of corruptions are almost never going to be exercised by random bit manipulation fuzzers like syzbot, but they are exactly the sort of thing a malicious attacker with some knowledge of how the filesystem works would look at.... Let's also address the elephant in the room: malicious attackers don't need to to exploit flaws in the filesystem metadata structure to trojan an unsuspecting user. i.e. We cannot detect changes to metadata that are within valid bounds and may be security sensitive - things like UIDs and GIDs, inode permissions, inode flags, link counts, symbolic links, etc. We also can't determine if the file data is unchanged, so it's easy to trojan the contents of an executable file on a filesystem image. IOWs, all the attacker needs to do is trojan an installer script on an application or device driver disk/image, and the user will run it as root themselves.... There are whole classes of malicious modifications that syzbot doesn't exercise and we cannot detect nor defend against at the filesystem level without changing the trust model the filesystem operates under. And if we change the trust model, we are now talking about on-disk format changes and using robust crypto for all the data and metadata in the filesystem. At which point, we may as well require a full disk encryption layer via dm-crypt.... If we say "filesystem is secure against untrusted input" then that is what users will expect us to provide. It will also means that every bug that syzbot might find will result in a high priority CVE, because any issue arising from untrusted input is a now a major system security issue. As such, I just don't see how "tested with syzbot" equates with "safe for untrusted use cases" whilst also reducing the impact of the problems that syzbot finds and reports... > Part 2: unmaintained file systems > > A lot of our file system drivers are either de facto or formally > unmaintained. If we want to move the kernel forward by finishing > API transitions (new mount API, buffer_head removal for the I/O path, > ->writepage removal, etc) these file systems need to change as well > and need some kind of testing. The easiest way forward would be > to remove everything that is not fully maintained, but that would > remove a lot of useful features. Linus has explicitly NACKed that approach. https://lore.kernel.org/linux-fsdevel/CAHk-=wg7DSNsHY6tWc=WLeqDBYtXges_12fFk1c+-No+fZ0xYQ@xxxxxxxxxxxxxx/ Which is a problem, because historically we've taken code into the kernel without requiring a maintainer, or the people who maintained the code have moved on, yet we don't have a policy for removing code that is slowly bit-rotting to uselessness. > E.g. the hfsplus driver is unmaintained despite collecting odd fixes. > It collects odd fixes because it is really useful for interoperating > with MacOS and it would be a pity to remove it. At the same time > it is impossible to test changes to hfsplus sanely as there is no > mkfs.hfsplus or fsck.hfsplus available for Linux. We used to have > one that was ported from the open source Darwin code drops, and > I managed to get xfstests to run on hfsplus with them, but this > old version doesn't compile on any modern Linux distribution and > new versions of the code aren't trivially portable to Linux. > > Do we have volunteers with old enough distros that we can list as > testers for this code? Do we have any other way to proceed? > > If we don't, are we just going to untested API changes to these > code bases, or keep the old APIs around forever? We do slowly remove device drivers and platforms as the hardware, developers and users disappear. We do also just change driver APIs in device drivers for hardware that no-one is actually able to test. The assumption is that if it gets broken during API changes, someone who needs it to work will fix it and send patches. That seems to be the historical model for removing unused/obsolete code from the kernel, so why should we treat unmaintained/obsolete filesystems any differently? i.e. Just change the API, mark it CONFIG_BROKEN until someone comes along and starts fixing it... -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
