On Tue, 2023-01-17 at 15:22 +0000, Chuck Lever III wrote:
>
> > On Jan 17, 2023, at 10:16 AM, Jason Gunthorpe <jgg@xxxxxxxxxx> wrote:
> >
> > On Tue, Nov 01, 2022 at 10:46:45AM -0400, Jeff Layton wrote:
> > > The filecache refcounting is a bit non-standard for something searchable
> > > by RCU, in that we maintain a sentinel reference while it's hashed. This
> > > in turn requires that we have to do things differently in the "put"
> > > depending on whether its hashed, which we believe to have led to races.
> > >
> > > There are other problems in here too. nfsd_file_close_inode_sync can end
> > > up freeing an nfsd_file while there are still outstanding references to
> > > it, and there are a number of subtle ToC/ToU races.
> > >
> > > Rework the code so that the refcount is what drives the lifecycle. When
> > > the refcount goes to zero, then unhash and rcu free the object.
> > >
> > > With this change, the LRU carries a reference. Take special care to
> > > deal with it when removing an entry from the list.
> > >
> > > Signed-off-by: Jeff Layton <jlayton@xxxxxxxxxx>
> >
> > Our test team is getting crashes that bisection pointed at this
> > patch. It seems like there are multiple parallel crash reports so the
> > whole thing is a mess to read:
> >
> > [ 875.548965] BUG: kernel NULL pointer dereference, address: 00000000000000d0
> > [ 875.548968] ------------[ cut here ]------------
> > [ 875.548972] refcount_t: underflow; use-after-free.
> > [ 875.548992] WARNING: CPU: 4 PID: 12145 at lib/refcount.c:28 refcount_warn_saturate+0xd8/0xe0
> > [ 875.549851] #PF: supervisor read access in kernel mode
> > [ 875.550158] Modules linked in:
> > [ 875.550752] #PF: error_code(0x0000) - not-present page
> > [ 875.551269] nfsd
> > [ 875.551878] PGD 0
> > [ 875.552069] iptable_raw
> > [ 875.552677] P4D 0
> > [ 875.552824] bonding mlx5_vfio_pci
> > [ 875.553095]
> > [ 875.553255] rdma_ucm ipip
> > [ 875.553525] Oops: 0000 [#1] SMP
> > [ 875.553733] tunnel4
> > [ 875.553941] CPU: 0 PID: 12147 Comm: nfsd Not tainted 6.1.0-rc7_ac3a2585f018 #1
> > [ 875.554109] ip_gre ib_umad
> > [ 875.554517] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> > [ 875.554656] nf_tables vfio_pci
> > [ 875.555508] RIP: 0010:vfs_setlease+0x27/0x70
> > [ 875.555695] vfio_pci_core vfio_virqfd
> > [ 875.557015] Code: ff ff 90 0f 1f 44 00 00 41 54 49 89 d4 55 48 89 fd 48 83 ec 10 48 85 d2 74 06 48 83 fe 02 75 1f 48 8b 45 28 4c 89 e2 48 89 ef <48> 8b 80 d0 00 00 00 48 85 c0 74 2c 48 83 c4 10 5d 41 5c ff e0 48
> > [ 875.557209] vfio_iommu_type1
> > [ 875.557406] RSP: 0018:ffff88810378bdb0 EFLAGS: 00010246
> > [ 875.557634] mlx5_ib
> > [ 875.558446]
> > [ 875.558628] vfio
> > [ 875.558862] RAX: 0000000000000000 RBX: ffff88824866c000 RCX: ffff88810378bdd8
> > [ 875.559006] ib_uverbs
> > [ 875.559092] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff88812560a200
> > [ 875.559218] ib_ipoib
> > [ 875.559557] RBP: ffff88812560a200 R08: ffff8881da5ecf00 R09: ffffffff824064e0
> > [ 875.559704] mlx5_core
> > [ 875.560021] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> > [ 875.560165] ip6_gre
> > [ 875.560488] R13: ffff8881da5ecf00 R14: ffff888110e62028 R15: ffff888110e621a0
> > [ 875.560634] gre
> > [ 875.560959] FS: 0000000000000000(0000) GS:ffff88852c800000(0000) knlGS:0000000000000000
> > [ 875.561108] ip6_tunnel
> > [ 875.561432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 875.561554] tunnel6
> > [ 875.561928] CR2: 00000000000000d0 CR3: 00000001ca27d001 CR4: 0000000000372eb0
> > [ 875.562084] geneve
> > [ 875.562349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 875.562493] nfnetlink_cttimeout
> > [ 875.562822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [ 875.562962] openvswitch
> > [ 875.563292] Call Trace:
> > [ 875.563298] <TASK>
> > [ 875.563503] nsh
> > [ 875.563839] destroy_unhashed_deleg+0x58/0xc0 [nfsd]
>
> We are aware of this failure mode. Actually this started well before
> that particular commit.
>
> Our problem has been that no one has been able to provide a reliable
> reproducer, so we can't figure out why it's happening. If you have a
> way to reproduce this failure reliably, can you capture a vmcore or
> enable KASAN and get a little more information?
>
It's possible that this crash may be related to the problem that was
fixed here:
commit 0b3a551fa58b4da941efeb209b3770868e2eddd7
Author: Jeff Layton <jlayton@xxxxxxxxxx>
Date: Thu Jan 5 14:55:56 2023 -0500
nfsd: fix handling of cached open files in nfsd4_open codepath
Unfortunately, that hasn't trickled into v6.1 kernels so far.
--
Jeff Layton <jlayton@xxxxxxxxxx>
