Hi -
I have a "standard" test of running the git regression suite with
many threads against an NFS mount. I found that with 6.5-rc, the
test stalled and several nfsd threads on the server were stuck
in D state.
I can reproduce this stall 100% with both an xfs and an ext4
export, so I bisected with both, and both bisects landed on the
same commit:
615939a2ae734e3e68c816d6749d1f5f79c62ab7 is the first bad commit
commit 615939a2ae734e3e68c816d6749d1f5f79c62ab7
Author: Christoph Hellwig <hch@xxxxxx>
Date: Fri May 19 06:40:48 2023 +0200
blk-mq: defer to the normal submission path for post-flush requests
Requests with the FUA bit on hardware without FUA support need a post
flush before returning to the caller, but they can still be sent using
the normal I/O path after initializing the flush-related fields and
end I/O handler.
Signed-off-by: Christoph Hellwig <hch@xxxxxx>
Reviewed-by: Bart Van Assche <bvanassche@xxxxxxx>
Link: https://lore.kernel.org/r/20230519044050.107790-6-hch@xxxxxx
Signed-off-by: Jens Axboe <axboe@xxxxxxxxx>
block/blk-flush.c | 11 +++++++++++
1 file changed, 11 insertions(+)
On system 1: the exports are on top of /dev/mapper and reside on
an "INTEL SSDSC2BA400G3" SATA device.
On system 2: the exports are on top of /dev/mapper and reside on
an "INTEL SSDSC2KB240G8" SATA device.
System 1 was where I discovered the stall. System 2 is where I ran
the bisects.
The call stacks vary a little. I've seen stalls in both the WRITE
and SETATTR paths. Here's a sample from system 1:
INFO: task nfsd:1237 blocked for more than 122 seconds.
Tainted: G W 6.4.0-08699-g9e268189cb14 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:nfsd state:D stack:0 pid:1237 ppid:2 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x78f/0x7db
schedule+0x93/0xc8
jbd2_log_wait_commit+0xb4/0xf4
? __pfx_autoremove_wake_function+0x10/0x10
jbd2_complete_transaction+0x85/0x97
ext4_fc_commit+0x118/0x70a
? _raw_spin_unlock+0x18/0x2e
? __mark_inode_dirty+0x282/0x302
ext4_write_inode+0x94/0x121
ext4_nfs_commit_metadata+0x72/0x7d
commit_inode_metadata+0x1f/0x31 [nfsd]
commit_metadata+0x26/0x33 [nfsd]
nfsd_setattr+0x2f2/0x30e [nfsd]
nfsd_create_setattr+0x4e/0x87 [nfsd]
nfsd4_open+0x604/0x8fa [nfsd]
nfsd4_proc_compound+0x4a8/0x5e3 [nfsd]
? nfs4svc_decode_compoundargs+0x291/0x2de [nfsd]
nfsd_dispatch+0xb3/0x164 [nfsd]
svc_process_common+0x3c7/0x53a [sunrpc]
? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
svc_process+0xc6/0xe3 [sunrpc]
nfsd+0xf2/0x18c [nfsd]
? __pfx_nfsd+0x10/0x10 [nfsd]
kthread+0x10d/0x115
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
--
Chuck Lever
