On 3/12/24 11:23 AM, Rik Theys wrote:
Hi,
On 3/12/24 17:43, Dai Ngo wrote:
On 3/12/24 4:37 AM, Jeff Layton wrote:
On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
Hi,
Since a few weeks our Rocky Linux 9 NFS server has periodically
logged hung nfsd tasks. The initial effect was that some clients
could no longer access the NFS server. This got worse and worse
(probably as more nfsd threads got blocked) and we had to restart
the server. Restarting the server also failed as the NFS server
service could no longer be stopped.
The initial kernel we noticed this behavior on was
kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
happened again on this newer kernel version:
[Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:10:08 2024] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
pid:8865 ppid:2 flags:0x00004000
[Mon Mar 11 14:10:08 2024] Call Trace:
[Mon Mar 11 14:10:08 2024] <TASK>
[Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
[Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
[nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:10:08 2024] ?
nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
[Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:10:08 2024] </TASK>
[Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more
than 122 seconds.
[Mon Mar 11 14:10:08 2024] Not tainted
5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:10:08 2024] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
pid:8866 ppid:2 flags:0x00004000
[Mon Mar 11 14:10:08 2024] Call Trace:
[Mon Mar 11 14:10:08 2024] <TASK>
[Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
[Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
[Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:10:08 2024] </TASK>
The above is repeated a few times, and then this warning is also
logged:
[Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
[Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver
nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm
ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
ibcrc32c dm_service_time dm_multipath intel_rapl_msr
intel_rapl_common intel_uncore_frequency
intel_uncore_frequency_common isst_if_common skx_edac nfit
libnvdimm ipmi_ssif x86_pkg_temp
_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas
rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios
drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache
jbd2 sd_mod sg lpfc
[Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
dm_region_hash dm_log dm_mod
[Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
tainted 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
R740/00WGD1, BIOS 2.20.1 09/13/2023
[Mon Mar 11 14:12:05 2024] RIP:
0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89
df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00
00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
[Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
00010246
[Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
ffff8ada51930900 RCX: 0000000000000024
[Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
ffff8ad582933c00 RDI: 0000000000002000
[Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
ffff9929e0bb7b48 R09: 0000000000000000
[Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
0000000000000000 R12: ffff8ad6f497c360
[Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
ffff8ae5942e0b10 R15: ffff8ad6f497c360
[Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
[Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
[Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
00000018e58de006 CR4: 00000000007706e0
[Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
[Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
[Mon Mar 11 14:12:05 2024] PKRU: 55555554
[Mon Mar 11 14:12:05 2024] Call Trace:
[Mon Mar 11 14:12:05 2024] <TASK>
[Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
[Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
[Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
[Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
[Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160
[nfsd]
[Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
[Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
[Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
[Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
[Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:12:05 2024] </TASK>
[Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
[Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0
pid:2451130 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks
nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
[Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
[Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
[Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
[Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
[Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
[Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
The above is the main task that I see in the cb workqueue. It's
trying to call rpc_shutdown_client, which is waiting for this:
wait_event_timeout(destroy_wait,
list_empty(&clnt->cl_tasks), 1*HZ);
...so basically waiting for the cl_tasks list to go empty. It
repeatedly
does a rpc_killall_tasks though, so possibly trying to kill this task?
18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd]
nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
I wonder why this task is on delayq. Could it be related to memory
shortage issue, or connection related problems?
Output of /proc/meminfo on the nfs server at time of the problem
would shed some light.
We don't have that anymore. I can check our monitoring host more
closely for more fine grained stats tomorrow, but when I look at the
sar statistics (see attachment) nothing special was going on memory or
network wise.
Thanks Rik for the info.
At 2:10 PM sar statistics shows:
kbmemfree: 1014880
kbavail: 170836368
kbmemused: 2160028
%memused: 1.10
kbcached: 140151288
Paging stats:
pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
02:10:00 PM 2577.67 491251.09 2247.01 0.00 2415029.61 75131.80 0.00 150276.28 200.02
The kbmemfree is pretty low and the caches consume large amount of memory.
The paging statistics also show lots of paging activities, 150276.28/s.
In the previous rpc_tasks.txt, it shows a RPC task is on the delayq waiting
to send the CB_RECALL_ANY. With this version of the kernel, the only time
CB_RECALL_ANY is sent is when the system is under memory pressure and the
nfsd shrinker task runs to free unused delegations.
Next time when this problem happens again, you can try to reclaim some
memory from the caches to see if it helps:
# echo 3 > /proc/sys/vm/drop_caches
-Dai
We start to get the cpu stall messages and the system load starts to
rise (starts around 2:10 PM). At 3:00 PM we restart the server as our
users can no longer work.
Looking at the stats, the cpu's were ~idle. The only thing that may be
related is that around 11:30 AM the write load (rx packets) starts to
get a lot higher than the read load (tx packets). This goes on for
hours (even after the server was restarted) and that workload was
later identified. It was a workload that was constantly rewriting a
statistics file.
Regards,
Rik
Currently there is only 1 active task allowed for the nfsd callback
workqueue at a time. If for some reasons a callback task is stuck in
the workqueue it will block all other callback tasks which can effect
multiple clients.
-Dai
Callbacks are soft RPC tasks though, so they should be easily killable.
