On Tue, Feb 6, 2024 at 4:52 AM Antonio SJ Musumeci <trapexit@xxxxxxxxxx> wrote:
>
> Hi,
>
> Anyone have users exporting a FUSE filesystem over NFS? Particularly
> from Proxmox (recent release, kernel 6.5.11)? I've gotten a number of
> reports recently from individuals who have such a setup and after some
> time (not easily reproducible, seems usually after software like Plex or
> Jellyfin do a scan of media or a backup process) starts returning EIO
> errors. Not just from NFS but also when trying to access the FUSE mount
> as well. One person noted that they had moved from Ubuntu 18.04 (kernel
> 4.15.0) to Proxmox and on Ubuntu had no problems with otherwise the same
> settings.
>
> I've not yet been able to reproduced this issue myself but wanted to see
> if anyone else has run into this. As far as I can tell from what users
> have reported the FUSE server is still running but isn't receiving most
> requests. I do see evidence of statfs calls coming through but nothing
> else. Though the straces I've received typically are after the issues start.
>
> In an effort to rule out the FUSE server... is there anything the server
> could do to cause the kernel to return EIO and not forward anything but
> statfs? Doesn't seem to matter if direct_io is enabled or attr/entry
> caching is used.
>
This could be the outcome of commit 15db16837a35 ("fuse: fix illegal
access to inode with reused nodeid") in kernel v5.14.
It is not an unintended regression - this behavior replaces what would
have been a potentially severe security violation with an EIO error.
As the commit says:
"...With current code, this situation will not be detected and an old fuse
dentry that used to point to an older generation real inode, can be used to
access a completely new inode, which should be accessed only via the new
dentry."
I have made this fix after seeing users get the content of another
file from the one that they opened in NFS!
libfuse commit 10ecd4f ("test/test_syscalls.c: check unlinked testfiles
at the end of the test") reproduces this problem in a test.
This test does not involve NFS export, but NFS export has higher
likelihood of exposing this issue.
I wonder if the FUSE filesystems that report the errors have
FUSE_EXPORT_SUPPORT capability?
Not that this capability guarantees anything wrt to this issue.
IMO, the root of all evil wrt NFS+FUSE is that LOOKUP is by ino
without generation with FUSE_EXPORT_SUPPORT, but worse
is that FUSE does not even require FUSE_EXPORT_SUPPORT
capability to export to NFS, but this is legacy FUSE behavior and
I am sure that many people export FUSE filesystems, as your
report proves.
There is now a proposal for opt-out of NFS export:
https://lore.kernel.org/linux-fsdevel/20240126072120.71867-1-jefflexu@xxxxxxxxxxxxxxxxx/
so there will be a way for a FUSE filesystem to prevent misuse.
Some practical suggestions for users running existing FUSE filesystems:
- Never export a FUSE filesystem with a fixed fsid
- Everytime one wants to export a FUSE filesystem generate
a oneshot fsid/uuid to use in exportfs
- Then restarting/re-exporting the FUSE filesystem will result in
ESTALE errors on NFS client, but not security violations and not EIO
errors
- This does not give full guarantee, unlinked inodes could still result
in EIO errors, as the libfuse test demonstrates
- The situation with NFSv4 is slightly better than with NFSv3, because
with NFSv3, an open file in the client does not keep the FUSE file
open and increases the chance of evicted FUSE inode for an open
NFS file
Thanks,
Amir.
