On 13 Nov 2023, at 9:52, Alexander Aring wrote:
> Hi nfs people,
>
> I use in my "fragile" kernel hacking development setup a nfsroot
> setup. I accidently turned on the kernel config
> CONFIG_WQ_CPU_INTENSIVE_REPORT. Now, sometimes I get:
>
> workqueue: nfs4_renew_state hogged CPU for >10000us 16 times, consider
> switching to WQ_UNBOUND
>
> and I just want to drop this mail so that nfs people are aware of
> this. I am not sure if nfs4_renew_state can be easily switched to
> WQ_UNBOUND and it really makes sense to do it.
>
> I also don't do any specific workload on nfs, as I mentioned I use it
> for nfsroot and I get several of those messages over time. I used nfs
> version 4.0 and those messages showed up. Jeff told me to switch to
> 4.2 and try it again, but I still get those messages.
>
> So here is my mail to start some kind of discussion about it. :)
This is the lease-maintenance function for v4.0. By the way, v4.0 has
problems that are easily fixed by using v4.1. Maybe we can start by trying
to figure out where nfs4_renew_state() is taking so much time. You can do
that with the function_grapher:
# echo nfs4_renew_state > /sys/kernel/debug/tracing/set_graph_function
# echo 3 > /sys/kernel/debug/tracing/max_graph_depth
# echo function_graph > /sys/kernel/debug/tracing/current_tracer
Wait one lease period (the default is 90 seconds), then
cat /sys/kernel/debug/tracing/trace
...
Cleanup those settings like this:
# echo nop > /sys/kernel/debug/tracing/current_tracer
# > /sys/kernel/debug/tracing/set_graph_function
# echo 0 > /sys/kernel/debug/tracing/max_graph_depth
My nfs4_renew_state() takes about 180us:
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) ! 135.125 us | nfs4_renew_state [nfsv4]();
4) | /* xprt_lookup_rqst: peer=[::1]:2049 xid=0x12cde0fc status=0 */
------------------------------------------
4) kworker-176058 => kworker-176218
------------------------------------------
4) | nfs4_renew_state [nfsv4]() {
4) | nfs_delegations_present [nfsv4]() {
4) 0.792 us | __rcu_read_lock();
4) 0.500 us | __rcu_read_unlock();
4) 5.834 us | }
4) | nfs4_get_renew_cred [nfsv4]() {
4) 3.709 us | nfs4_get_machine_cred [nfsv4]();
4) 5.000 us | }
4) | nfs4_proc_async_renew [nfsv4]() {
4) + 27.167 us | kmalloc_trace();
4) + 97.167 us | rpc_call_async [sunrpc]();
4) ! 151.375 us | }
4) | nfs_expire_unreferenced_delegations [nfsv4]() {
4) 0.458 us | __rcu_read_lock();
4) 0.542 us | __rcu_read_unlock();
4) 2.541 us | }
4) ! 178.792 us | }
------------------------------------------
4) kworker-176218 => kworker-175836
------------------------------------------
Ben
