Trond,
I wonder if you are aware of this or not.
This week I ran a lot of xfstests with hammerspace and other servers
without any issues and just now seeing this oops (after rebuilding
from a base of your testing branch at 0abb8895b065). I then re-built
with just your testing branch and got the same oops. Same test passes
on 5.14.0-rc4 (vanilla), as well as previous kernels I used at
BakeAthon with the fscache and readahead patches only. It reliably
panics for me so let me know if you want any more info or a
reproduction with tracepoints, etc. FYI, I don't think the server
matters because I can also reproduce with a rhel8 server
(kernel-4.18.0-305.19.1.el8_4) and I can also just run 'generic/005'
directly - previous tests don't matter.
[ 15.767423] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 28.614447] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver
Registering...
[ 30.024616] run fstests generic/001 at 2021-10-09 09:01:14
[ 37.188167] run fstests generic/002 at 2021-10-09 09:01:21
[ 38.372767] run fstests generic/003 at 2021-10-09 09:01:23
[ 38.713218] run fstests generic/004 at 2021-10-09 09:01:23
[ 39.065705] run fstests generic/005 at 2021-10-09 09:01:23
[ 39.799076] general protection fault, probably for non-canonical
address 0xffe826e8e8e7897c: 0000 [#1] SMP PTI
[ 39.805058] CPU: 0 PID: 6213 Comm: rm Kdump: loaded Not tainted
5.15.0-rc4-trond-testing+ #76
[ 39.808300] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS 1.14.0-4.fc34 04/01/2014
[ 39.810819] RIP: 0010:__mutex_lock.constprop.0+0x97/0x3e0
[ 39.812438] Code: 65 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 48 8b
04 25 c0 7b 01 00 48 8b 00 a8 08 75 1d 49 8b 06 48 83 e0 f8 0f 84 79
02 00 00 <8b> 50 34 85 d2 0f 85 5c 02 00 00 e8 59 8f 55 ff 65 48 8b 04
25 c0
[ 39.817418] RSP: 0018:ffffbb8e4558bcd0 EFLAGS: 00010286
[ 39.818546] RAX: ffe826e8e8e78948 RBX: ffffbb8e4558bd70 RCX: ffff932e89300000
[ 39.820087] RDX: ffff932e89300000 RSI: ffe826e8e8e78948 RDI: ffff932eae01edf0
[ 39.821583] RBP: ffffbb8e4558bd28 R08: 0000000000000001 R09: ffffbb8e4558bca0
[ 39.823091] R10: 000000000000001d R11: ffffffffffffcfcf R12: 0000000000000000
[ 39.824796] R13: 0000000000000000 R14: ffff932eae01edf0 R15: 0000000000000000
[ 39.826318] FS: 00007fac7df09740(0000) GS:ffff932ff7c00000(0000)
knlGS:0000000000000000
[ 39.828037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 39.829275] CR2: 0000555f9b7e2018 CR3: 000000011dc96005 CR4: 0000000000770ef0
[ 39.830787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 39.832275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 39.833756] PKRU: 55555554
[ 39.834377] Call Trace:
[ 39.836219] ? security_inode_permission+0x30/0x50
[ 39.837365] nfs_scan_commit+0x36/0xa0 [nfs]
[ 39.838367] __nfs_commit_inode+0xf8/0x160 [nfs]
[ 39.839417] nfs_wb_all+0xa6/0xf0 [nfs]
[ 39.840309] nfs4_inode_return_delegation+0x58/0x80 [nfsv4]
[ 39.841554] nfs4_proc_remove+0xd1/0xe0 [nfsv4]
[ 39.842589] nfs_unlink+0xec/0x2d0 [nfs]
[ 39.843461] vfs_unlink+0x113/0x230
[ 39.844245] do_unlinkat+0x170/0x280
[ 39.845040] __x64_sys_unlinkat+0x33/0x60
[ 39.845922] do_syscall_64+0x3b/0x90
[ 39.846726] entry_SYSCALL_64_after_hwframe+0x44/0xae
# ./scripts/faddr2line fs/nfs/nfs.ko nfs_scan_commit+0x36
nfs_scan_commit+0x36/0xa0:
nfs_scan_commit at /mnt/build/kernel/fs/nfs/write.c:1078
(inlined by) nfs_scan_commit at /mnt/build/kernel/fs/nfs/write.c:1070
1060 /*
1061 * nfs_scan_commit - Scan an inode for commit requests
1062 * @inode: NFS inode to scan
1063 * @dst: mds destination list
1064 * @cinfo: mds and ds lists of reqs ready to commit
1065 *
1066 * Moves requests from the inode's 'commit' request list.
1067 * The requests are *not* checked to ensure that they form a
contiguous set.
1068 */
1069 int
1070 nfs_scan_commit(struct inode *inode, struct list_head *dst,
1071 struct nfs_commit_info *cinfo)
1072 {
1073 int ret = 0;
1074
1075 if (!atomic_long_read(&cinfo->mds->ncommit))
1076 return 0;
1077 mutex_lock(&NFS_I(cinfo->inode)->commit_mutex);
1078 if (atomic_long_read(&cinfo->mds->ncommit) > 0) {
1079 const int max = INT_MAX;
1080
1081 ret = nfs_scan_commit_list(&cinfo->mds->list, dst,
1082 cinfo, max);
1083 ret += pnfs_scan_commit_lists(inode, cinfo, max - ret);
1084 }
1085 mutex_unlock(&NFS_I(cinfo->inode)->commit_mutex);
1086 return ret;
1087 }
