On Mon, Dec 23, 2024 at 5:37 PM Yafang Shao <laoar.shao@xxxxxxxxx> wrote: > > We recently encountered an XFS deadlock issue, which is a known problem > resolved in the upstream kernel [0]. During the analysis of this issue, I > observed that a kernel thread in the TASK_WAKEKILL state could not be > detected as a hung task by the hung_task detector. The details are as > follows: > > Using the following command, I identified nine tasks stuck in the D state: > > $ ps -eLo state,comm,tid,wchan | grep ^D > D java 4177339 xfs_buf_lock > D kworker/93:3+xf 3025535 xfs_buf_lock > D kworker/87:0+xf 3426612 xfs_extent_busy_flush > D kworker/85:0+xf 3479378 xfs_buf_lock > D kworker/91:1+xf 3584478 xfs_buf_lock > D kworker/80:3+xf 3655680 xfs_buf_lock > D kworker/89:0+xf 3671691 xfs_buf_lock > D kworker/84:1+xf 3708397 xfs_buf_lock > D kworker/81:1+xf 4005763 xfs_buf_lock > > However, the hung_task detector only reported eight of these tasks: > > [3108840.650652] INFO: task java:4177339 blocked for more than 247779 seconds. > [3108840.654197] INFO: task kworker/93:3:3025535 blocked for more than 248427 seconds. > [3108840.657711] INFO: task kworker/85:0:3479378 blocked for more than 247836 seconds. > [3108840.661483] INFO: task kworker/91:1:3584478 blocked for more than 249638 seconds. > [3108840.664871] INFO: task kworker/80:3:3655680 blocked for more than 249638 seconds. > [3108840.668495] INFO: task kworker/89:0:3671691 blocked for more than 249047 seconds. > [3108840.672418] INFO: task kworker/84:1:3708397 blocked for more than 247836 seconds. > [3108840.676175] INFO: task kworker/81:1:4005763 blocked for more than 247836 seconds. > > Task 3426612, although in the D state, was not reported as a hung task. > > I confirmed that task 3426612 remained in the D (disk sleep) state and > experienced no context switches over a long period: > > $ cat /proc/3426612/status | grep -E "State:|ctxt_switches:"; \ > sleep 60; echo "----"; \ > cat /proc/3426612/status | grep -E "State:|ctxt_switches:" > State: D (disk sleep) > voluntary_ctxt_switches: 7516 > nonvoluntary_ctxt_switches: 0 > ---- > State: D (disk sleep) > voluntary_ctxt_switches: 7516 > nonvoluntary_ctxt_switches: 0 > > The system's hung_task detector settings were configured as follows: > > kernel.hung_task_timeout_secs = 28 > kernel.hung_task_warnings = -1 > > The issue lies in the handling of task state in the XFS code. Specifically, > the thread in question (3426612) was set to the TASK_KILLABLE state in > xfs_extent_busy_flush(): > > xfs_extent_busy_flush > prepare_to_wait(&pag->pagb_wait, &wait, TASK_KILLABLE); > > When a task is in the TASK_WAKEKILL state (a subset of TASK_KILLABLE), the > hung_task detector ignores it, as it assumes such tasks can be terminated. > However, in this case, the kernel thread cannot be killed, meaning it > effectively becomes a hung task. > > To address this issue, the hung_task detector should report the kthreads in > the TASK_WAKEKILL state. > > Link: https://lore.kernel.org/linux-xfs/20230620002021.1038067-5-david@xxxxxxxxxxxxx/ [0] > Signed-off-by: Yafang Shao <laoar.shao@xxxxxxxxx> > Cc: Dave Chinner <david@xxxxxxxxxxxxx> > --- > kernel/hung_task.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/kernel/hung_task.c b/kernel/hung_task.c > index c18717189f32..ed63fd84ce2e 100644 > --- a/kernel/hung_task.c > +++ b/kernel/hung_task.c > @@ -220,8 +220,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > */ > state = READ_ONCE(t->__state); > if ((state & TASK_UNINTERRUPTIBLE) && > + (t->flags & PF_KTHREAD || > !(state & TASK_WAKEKILL) && > - !(state & TASK_NOLOAD)) > + !(state & TASK_NOLOAD))) > check_hung_task(t, timeout); > } > unlock: > -- > 2.43.5 > Please disregard this. There may be multiple hung tasks in the TASK_IDLE state. I will send a new one. -- Regards Yafang
