Hello Johannes, If you have time, could you please take a look at this? I do not have any quick or reliable way to reproduce this, but since one of the threads is blocked on a debugfs completion, then I'm suspicious it may be some other bug in the area you previously (mostly?) fixed. Maybe because the kworkers are blocked on ipv6 stuff so whatever completion work we are waiting on will never complete? gdb) l *(wiphy_locked_debugfs_read+0x114) 0x713e4 is in wiphy_locked_debugfs_read (/home/greearb/git/linux-6.7.dev.y/net/wireless/debugfs.c:182). 177 wiphy_work_init(&work.work, wiphy_locked_debugfs_read_work); 178 wiphy_work_queue(wiphy, &work.work); 179 180 debugfs_enter_cancellation(file, &cancellation); 181 wait_for_completion(&work.completion); 182 debugfs_leave_cancellation(file, &cancellation); 183 184 if (work.ret < 0) 185 return work.ret; 186 (gdb) [905261.271981] INFO: task systemd-resolve:948 blocked for more than 180 seconds. [905261.277834] Tainted: G O 6.7.3+ #34 [905261.281939] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.288474] task:systemd-resolve state:D stack:0 pid:948 tgid:948 ppid:1 flags:0x00000002 [905261.288477] Call Trace: [905261.288478] <TASK> [905261.288481] __schedule+0x377/0xa10 [905261.288486] ? _copy_to_user+0x20/0x30 [905261.288507] schedule+0x29/0xc0 [905261.288509] schedule_preempt_disabled+0xc/0x20 [905261.288511] __mutex_lock.constprop.0+0x34d/0x760 [905261.288514] do_ip_setsockopt+0x4a/0x1150 [905261.288518] ? ___sys_recvmsg+0x82/0xc0 [905261.288520] ip_setsockopt+0x2a/0xb0 [905261.288523] do_sock_setsockopt+0xa8/0x170 [905261.288527] __sys_setsockopt+0x85/0xd0 [905261.288529] __x64_sys_setsockopt+0x16/0x20 [905261.288530] do_syscall_64+0x3b/0x110 [905261.288533] entry_SYSCALL_64_after_hwframe+0x46/0x4e [905261.288537] RIP: 0033:0x7f228251389e [905261.288539] RSP: 002b:00007ffd4466fd38 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 [905261.288541] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f228251389e [905261.288542] RDX: 0000000000000024 RSI: 0000000000000000 RDI: 000000000000000a [905261.288543] RBP: 000000000000000a R08: 000000000000000c R09: 8e92374df5b14b7b [905261.288544] R10: 00007ffd4466fd44 R11: 0000000000000246 R12: 00007ffd4466fd44 [905261.288546] R13: 0000000000000000 R14: 0000561411a0ae00 R15: 0000561411a0ae00 [905261.288548] </TASK> [905261.288556] INFO: task ntpd:1435 blocked for more than 180 seconds. [905261.293524] Tainted: G O 6.7.3+ #34 [905261.297632] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.304168] task:ntpd state:D stack:0 pid:1435 tgid:1435 ppid:1 flags:0x00000006 [905261.304170] Call Trace: [905261.304171] <TASK> [905261.304172] __schedule+0x377/0xa10 [905261.304174] schedule+0x29/0xc0 [905261.304175] schedule_preempt_disabled+0xc/0x20 [905261.304177] __mutex_lock.constprop.0+0x34d/0x760 [905261.304179] __netlink_dump_start+0x6c/0x280 [905261.304196] ? rtnl_fill_ifinfo+0x1510/0x1510 [905261.304198] rtnetlink_rcv_msg+0x267/0x3b0 [905261.304202] ? rtnl_fill_ifinfo+0x1510/0x1510 [905261.304203] ? rtnl_calcit.isra.0+0x120/0x120 [905261.304205] netlink_rcv_skb+0x4f/0x100 [905261.304207] netlink_unicast+0x18b/0x280 [905261.304209] netlink_sendmsg+0x24b/0x4c0 [905261.304211] __sock_sendmsg+0x33/0x60 [905261.304212] __sys_sendto+0xec/0x130 [905261.304214] ? __sys_socket+0xc1/0xf0 [905261.304216] __x64_sys_sendto+0x1b/0x20 [905261.304217] do_syscall_64+0x3b/0x110 [905261.304218] entry_SYSCALL_64_after_hwframe+0x46/0x4e [905261.304221] RIP: 0033:0x7f438bb13834 [905261.304222] RSP: 002b:00007fff186e10f0 EFLAGS: 00000293 ORIG_RAX: 000000000000002c [905261.304223] RAX: ffffffffffffffda RBX: 00007fff186e2180 RCX: 00007f438bb13834 [905261.304225] RDX: 0000000000000014 RSI: 00007fff186e21c0 RDI: 0000000000000004 [905261.304225] RBP: 0000000000000000 R08: 00007fff186e2180 R09: 000000000000000c [905261.304226] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fff186e21c0 [905261.304227] R13: 000055fd78123140 R14: 00007fff186e1130 R15: 00007fff186e2280 [905261.304229] </TASK> [905261.304239] INFO: task abrt-applet:2433 blocked for more than 180 seconds. [905261.309820] Tainted: G O 6.7.3+ #34 [905261.313926] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.320462] task:abrt-applet state:D stack:0 pid:2433 tgid:2433 ppid:2046 flags:0x00000002 [905261.320465] Call Trace: [905261.320466] <TASK> [905261.320467] __schedule+0x377/0xa10 [905261.320469] ? poll_freewait+0x40/0xa0 [905261.320472] schedule+0x29/0xc0 [905261.320487] schedule_preempt_disabled+0xc/0x20 [905261.320489] __mutex_lock.constprop.0+0x34d/0x760 [905261.320492] __netlink_dump_start+0x6c/0x280 [905261.320494] ? rtnl_mdb_dump+0x160/0x160 [905261.320497] rtnetlink_rcv_msg+0x267/0x3b0 [905261.320499] ? rtnl_mdb_dump+0x160/0x160 [905261.320501] ? rtnl_calcit.isra.0+0x120/0x120 [905261.320504] netlink_rcv_skb+0x4f/0x100 [905261.320506] netlink_unicast+0x18b/0x280 [905261.320507] netlink_sendmsg+0x24b/0x4c0 [905261.320509] __sock_sendmsg+0x33/0x60 [905261.320510] ? sockfd_lookup_light+0xd/0x60 [905261.320513] __sys_sendto+0xec/0x130 [905261.320515] ? eventfd_read+0xc9/0x200 [905261.320517] ? eventfd_write+0xc0/0x1d0 [905261.320519] ? ktime_get_ts64+0x44/0xe0 [905261.320523] ? switch_fpu_return+0x47/0xc0 [905261.320526] __x64_sys_sendto+0x1b/0x20 [905261.320527] do_syscall_64+0x3b/0x110 [905261.320529] entry_SYSCALL_64_after_hwframe+0x46/0x4e [905261.320532] RIP: 0033:0x7f01c15136cc [905261.320533] RSP: 002b:00007fff04b4a210 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [905261.320535] RAX: ffffffffffffffda RBX: 000055b502c7f3d0 RCX: 00007f01c15136cc [905261.320536] RDX: 0000000000000014 RSI: 00007fff04b4a290 RDI: 0000000000000003 [905261.320537] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [905261.320538] R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000014 [905261.320539] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [905261.320540] </TASK> [905261.320542] INFO: task xdg-desktop-por:2485 blocked for more than 180 seconds. [905261.326467] Tainted: G O 6.7.3+ #34 [905261.330577] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.337114] task:xdg-desktop-por state:D stack:0 pid:2485 tgid:2485 ppid:1742 flags:0x00000002 [905261.337116] Call Trace: [905261.337117] <TASK> [905261.337118] __schedule+0x377/0xa10 [905261.337120] ? poll_freewait+0x40/0xa0 [905261.337122] schedule+0x29/0xc0 [905261.337138] schedule_preempt_disabled+0xc/0x20 [905261.337139] __mutex_lock.constprop.0+0x34d/0x760 [905261.337142] __netlink_dump_start+0x6c/0x280 [905261.337143] ? rtnl_mdb_dump+0x160/0x160 [905261.337146] rtnetlink_rcv_msg+0x267/0x3b0 [905261.337148] ? rtnl_mdb_dump+0x160/0x160 [905261.337150] ? rtnl_calcit.isra.0+0x120/0x120 [905261.337153] netlink_rcv_skb+0x4f/0x100 [905261.337155] netlink_unicast+0x18b/0x280 [905261.337156] netlink_sendmsg+0x24b/0x4c0 [905261.337158] __sock_sendmsg+0x33/0x60 [905261.337159] ? sockfd_lookup_light+0xd/0x60 [905261.337162] __sys_sendto+0xec/0x130 [905261.337163] ? eventfd_read+0xc9/0x200 [905261.337165] ? eventfd_write+0xc0/0x1d0 [905261.337167] ? ktime_get_ts64+0x44/0xe0 [905261.337170] ? switch_fpu_return+0x47/0xc0 [905261.337172] __x64_sys_sendto+0x1b/0x20 [905261.337173] do_syscall_64+0x3b/0x110 [905261.337174] entry_SYSCALL_64_after_hwframe+0x46/0x4e [905261.337178] RIP: 0033:0x7f21443136cc [905261.337179] RSP: 002b:00007ffc9a315160 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [905261.337180] RAX: ffffffffffffffda RBX: 0000561a0dfc43d0 RCX: 00007f21443136cc [905261.337181] RDX: 0000000000000014 RSI: 00007ffc9a3151e0 RDI: 0000000000000008 [905261.337182] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [905261.337183] R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000014 [905261.337184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [905261.337185] </TASK> [905261.337190] INFO: task gnuserver:3266 blocked for more than 180 seconds. [905261.342597] Tainted: G O 6.7.3+ #34 [905261.346705] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.353238] task:gnuserver state:D stack:0 pid:3266 tgid:3266 ppid:3218 flags:0x00000002 [905261.353240] Call Trace: [905261.353241] <TASK> [905261.353241] __schedule+0x377/0xa10 [905261.353243] schedule+0x29/0xc0 [905261.353245] schedule_preempt_disabled+0xc/0x20 [905261.353247] __mutex_lock.constprop.0+0x34d/0x760 [905261.353249] dev_ethtool+0x9e/0x29a0 [905261.353267] ? atime_needs_update+0x93/0x110 [905261.353270] ? touch_atime+0x15/0x110 [905261.353273] ? filemap_read+0x2e2/0x300 [905261.353277] dev_ioctl+0x256/0x480 [905261.353280] sock_do_ioctl+0x8e/0xd0 [905261.353281] sock_ioctl+0x157/0x2a0 [905261.353283] __x64_sys_ioctl+0x87/0xc0 [905261.353285] do_syscall_64+0x3b/0x110 [905261.353287] entry_SYSCALL_64_after_hwframe+0x46/0x4e [905261.353289] RIP: 0033:0x7f40e5b0774f [905261.353290] RSP: 002b:00007fffca032a30 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [905261.353291] RAX: ffffffffffffffda RBX: 00007fffca032b10 RCX: 00007f40e5b0774f [905261.353292] RDX: 00007fffca033190 RSI: 0000000000008946 RDI: 0000000000000008 [905261.353293] RBP: 00007fffca033190 R08: 000000000000000a R09: 00224f4e223d4c54 [905261.353294] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000008 [905261.353295] R13: 0000000000000008 R14: 00007fffca033240 R15: 0000000000000000 [905261.353297] </TASK> [905261.353300] INFO: task btserver:3593 blocked for more than 180 seconds. [905261.358620] Tainted: G O 6.7.3+ #34 [905261.362729] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.369263] task:btserver state:D stack:0 pid:3593 tgid:3593 ppid:2866 flags:0x00000002 [905261.369265] Call Trace: [905261.369266] <TASK> [905261.369267] __schedule+0x377/0xa10 [905261.369268] ? sched_clock+0xc/0x20 [905261.369270] schedule+0x29/0xc0 [905261.369271] schedule_timeout+0x136/0x150 [905261.369274] wait_for_completion+0x7d/0x130 [905261.369292] wiphy_locked_debugfs_read+0x114/0x180 [cfg80211] [905261.369363] ? wiphy_locked_debugfs_write_work+0x30/0x30 [cfg80211] [905261.369400] ? rts_threshold_read+0x80/0x80 [cfg80211] [905261.369438] ? ieee80211_link_sta_debugfs_drv_remove+0x40/0x40 [mac80211] [905261.369516] ieee80211_if_read_sdata+0x55/0x80 [mac80211] [905261.369553] ? ieee80211_link_sta_debugfs_drv_remove+0x40/0x40 [mac80211] [905261.369587] ? ieee80211_if_fmt_dot11MeshMaxRetries+0x20/0x20 [mac80211] [905261.369621] full_proxy_read+0x4e/0x70 [905261.369625] vfs_read+0xa3/0x2c0 [905261.369628] ksys_read+0x5e/0xe0 [905261.369630] do_syscall_64+0x3b/0x110 [905261.369631] entry_SYSCALL_64_after_hwframe+0x46/0x4e [905261.369633] RIP: 0033:0x7f27db101b92 [905261.369634] RSP: 002b:00007ffffe00d918 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [905261.369636] RAX: ffffffffffffffda RBX: 00007ffffe00da20 RCX: 00007f27db101b92 [905261.369637] RDX: 0000000000001fff RSI: 0000000009dae8f0 RDI: 0000000000000012 [905261.369638] RBP: 0000000000001fff R08: 0000000000002000 R09: 0000000000000000 [905261.369639] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000009dae8f0 [905261.369640] R13: 00007ffffe00da88 R14: 0000000003971712 R15: 0000000000000001 [905261.369641] </TASK> [905261.369667] INFO: task kworker/6:1:2567676 blocked for more than 180 seconds. [905261.375519] Tainted: G O 6.7.3+ #34 [905261.379656] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.386240] task:kworker/6:1 state:D stack:0 pid:2567676 tgid:2567676 ppid:2 flags:0x00004000 [905261.386243] Workqueue: ipv6_addrconf addrconf_dad_work [905261.386247] Call Trace: [905261.386248] <TASK> [905261.386249] __schedule+0x377/0xa10 [905261.386251] ? ttwu_do_activate+0x58/0x1d0 [905261.386255] schedule+0x29/0xc0 [905261.386269] schedule_preempt_disabled+0xc/0x20 [905261.386270] __mutex_lock.constprop.0+0x34d/0x760 [905261.386273] addrconf_dad_work+0x39/0x590 [905261.386275] ? __schedule+0x37f/0xa10 [905261.386276] process_one_work+0x15d/0x330 [905261.386291] worker_thread+0x2e8/0x400 [905261.386293] ? rescuer_thread+0x400/0x400 [905261.386294] kthread+0xdc/0x110 [905261.386298] ? kthread_complete_and_exit+0x20/0x20 [905261.386300] ret_from_fork+0x28/0x40 [905261.386302] ? kthread_complete_and_exit+0x20/0x20 [905261.386310] ret_from_fork_asm+0x11/0x20 [905261.386315] </TASK> [905261.386317] INFO: task kworker/7:1:2579027 blocked for more than 180 seconds. [905261.392164] Tainted: G O 6.7.3+ #34 [905261.396313] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.402878] task:kworker/7:1 state:D stack:0 pid:2579027 tgid:2579027 ppid:2 flags:0x00004000 [905261.402881] Workqueue: ipv6_addrconf addrconf_dad_work [905261.402884] Call Trace: [905261.402884] <TASK> [905261.402885] __schedule+0x377/0xa10 [905261.402887] schedule+0x29/0xc0 [905261.402889] schedule_preempt_disabled+0xc/0x20 [905261.402891] __mutex_lock.constprop.0+0x34d/0x760 [905261.402893] addrconf_dad_work+0x39/0x590 [905261.402895] ? __schedule+0x37f/0xa10 [905261.402896] process_one_work+0x15d/0x330 [905261.402898] worker_thread+0x2e8/0x400 [905261.402906] ? rescuer_thread+0x400/0x400 [905261.402908] kthread+0xdc/0x110 [905261.402911] ? kthread_complete_and_exit+0x20/0x20 [905261.402913] ret_from_fork+0x28/0x40 [905261.402915] ? kthread_complete_and_exit+0x20/0x20 [905261.402917] ret_from_fork_asm+0x11/0x20 [905261.402920] </TASK> [905261.402921] INFO: task kworker/u16:17:2582370 blocked for more than 180 seconds. [905261.409024] Tainted: G O 6.7.3+ #34 [905261.413157] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.419735] task:kworker/u16:17 state:D stack:0 pid:2582370 tgid:2582370 ppid:2 flags:0x00004000 [905261.419738] Workqueue: events_unbound cfg80211_wiphy_work [cfg80211] [905261.419788] Call Trace: [905261.419788] <TASK> [905261.419789] __schedule+0x377/0xa10 [905261.419792] schedule+0x29/0xc0 [905261.419793] schedule_preempt_disabled+0xc/0x20 [905261.419795] __mutex_lock.constprop.0+0x34d/0x760 [905261.419797] cfg80211_wiphy_work+0x15/0xb0 [cfg80211] [905261.419827] process_one_work+0x15d/0x330 [905261.419829] worker_thread+0x2e8/0x400 [905261.419831] ? rescuer_thread+0x400/0x400 [905261.419832] kthread+0xdc/0x110 [905261.419834] ? kthread_complete_and_exit+0x20/0x20 [905261.419854] ret_from_fork+0x28/0x40 [905261.419856] ? kthread_complete_and_exit+0x20/0x20 [905261.419859] ret_from_fork_asm+0x11/0x20 [905261.419862] </TASK> [905261.419864] INFO: task kworker/0:1:2591382 blocked for more than 180 seconds. [905261.425716] Tainted: G O 6.7.3+ #34 [905261.429854] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [905261.436399] task:kworker/0:1 state:D stack:0 pid:2591382 tgid:2591382 ppid:2 flags:0x00004000 [905261.436403] Workqueue: ipv6_addrconf addrconf_dad_work [905261.436406] Call Trace: [905261.436421] <TASK> [905261.436422] __schedule+0x377/0xa10 [905261.436424] schedule+0x29/0xc0 [905261.436425] schedule_preempt_disabled+0xc/0x20 [905261.436427] __mutex_lock.constprop.0+0x34d/0x760 [905261.436429] addrconf_dad_work+0x39/0x590 [905261.436431] ? __schedule+0x37f/0xa10 [905261.436433] process_one_work+0x15d/0x330 [905261.436435] worker_thread+0x2e8/0x400 [905261.436436] ? rescuer_thread+0x400/0x400 [905261.436437] kthread+0xdc/0x110 [905261.436440] ? kthread_complete_and_exit+0x20/0x20 [905261.436442] ret_from_fork+0x28/0x40 [905261.436443] ? kthread_complete_and_exit+0x20/0x20 [905261.436446] ret_from_fork_asm+0x11/0x20 [905261.436448] </TASK> [905261.436449] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings Thanks, Ben -- Ben Greear <greearb@xxxxxxxxxxxxxxx> Candela Technologies Inc http://www.candelatech.com
