On Thu, Jul 28, 2022 at 10:43:27AM +0800, Zhen Lei wrote:
> For architectures that do not support NMI interrupts, registers is not
> printed when rcu stall is self-detected. However, this information is
> useful for analyzing the root cause of the fault. Fortunately, the rcu
> stall is always detected in the tick interrupt handler. So we can take
> it through get_irq_regs() and display it through show_regs(). Further,
> show_regs() unwind the call trace based on 'regs', the worthless call
> trace associated with tick handling will be omitted, this helps us to
> focus more on the problem.
>
> This is an example on arm64:
> [ 27.501721] rcu: INFO: rcu_preempt self-detected stall on CPU
> [ 27.502238] rcu: 0-....: (1250 ticks this GP) idle=4f7/1/0x4000000000000000 softirq=2594/2594 fqs=619
> [ 27.502632] (t=1251 jiffies g=2989 q=29 ncpus=4)
> [ 27.503845] CPU: 0 PID: 306 Comm: test0 Not tainted 5.19.0-rc7-00009-g1c1a6c29ff99-dirty #46
> [ 27.504732] Hardware name: linux,dummy-virt (DT)
> [ 27.504947] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 27.504998] pc : arch_counter_read+0x18/0x24
> [ 27.505301] lr : arch_counter_read+0x18/0x24
> [ 27.505328] sp : ffff80000b29bdf0
> [ 27.505345] x29: ffff80000b29bdf0 x28: 0000000000000000 x27: 0000000000000000
> [ 27.505475] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
> [ 27.505553] x23: 0000000000001f40 x22: ffff800009849c48 x21: 000000065f871ae0
> [ 27.505627] x20: 00000000000025ec x19: ffff80000a6eb300 x18: ffffffffffffffff
> [ 27.505654] x17: 0000000000000001 x16: 0000000000000000 x15: ffff80000a6d0296
> [ 27.505681] x14: ffffffffffffffff x13: ffff80000a29bc18 x12: 0000000000000426
> [ 27.505709] x11: 0000000000000162 x10: ffff80000a2f3c18 x9 : ffff80000a29bc18
> [ 27.505736] x8 : 00000000ffffefff x7 : ffff80000a2f3c18 x6 : 00000000759bd013
> [ 27.505761] x5 : 01ffffffffffffff x4 : 0002dc6c00000000 x3 : 0000000000000017
> [ 27.505787] x2 : 00000000000025ec x1 : ffff80000b29bdf0 x0 : 0000000075a30653
> [ 27.505937] Call trace:
> [ 27.506002] arch_counter_read+0x18/0x24
> [ 27.506171] ktime_get+0x48/0xa0
> [ 27.506207] test_task+0x70/0xf0
> [ 27.506227] kthread+0x10c/0x110
> [ 27.506243] ret_from_fork+0x10/0x20
>
> The old output is as follows:
> [ 27.944550] rcu: INFO: rcu_preempt self-detected stall on CPU
> [ 27.944980] rcu: 0-....: (1249 ticks this GP) idle=cbb/1/0x4000000000000000 softirq=2610/2610 fqs=614
> [ 27.945407] (t=1251 jiffies g=2681 q=28 ncpus=4)
> [ 27.945731] Task dump for CPU 0:
> [ 27.945844] task:test0 state:R running task stack: 0 pid: 306 ppid: 2 flags:0x0000000a
> [ 27.946073] Call trace:
> [ 27.946151] dump_backtrace.part.0+0xc8/0xd4
> [ 27.946378] show_stack+0x18/0x70
> [ 27.946405] sched_show_task+0x150/0x180
> [ 27.946427] dump_cpu_task+0x44/0x54
> [ 27.947193] rcu_dump_cpu_stacks+0xec/0x130
> [ 27.947212] rcu_sched_clock_irq+0xb18/0xef0
> [ 27.947231] update_process_times+0x68/0xac
> [ 27.947248] tick_sched_handle+0x34/0x60
> [ 27.947266] tick_sched_timer+0x4c/0xa4
> [ 27.947281] __hrtimer_run_queues+0x178/0x360
> [ 27.947295] hrtimer_interrupt+0xe8/0x244
> [ 27.947309] arch_timer_handler_virt+0x38/0x4c
> [ 27.947326] handle_percpu_devid_irq+0x88/0x230
> [ 27.947342] generic_handle_domain_irq+0x2c/0x44
> [ 27.947357] gic_handle_irq+0x44/0xc4
> [ 27.947376] call_on_irq_stack+0x2c/0x54
> [ 27.947415] do_interrupt_handler+0x80/0x94
> [ 27.947431] el1_interrupt+0x34/0x70
> [ 27.947447] el1h_64_irq_handler+0x18/0x24
> [ 27.947462] el1h_64_irq+0x64/0x68 <--- the above backtrace is worthless
> [ 27.947474] arch_counter_read+0x18/0x24
> [ 27.947487] ktime_get+0x48/0xa0
> [ 27.947501] test_task+0x70/0xf0
> [ 27.947520] kthread+0x10c/0x110
> [ 27.947538] ret_from_fork+0x10/0x20
>
> Signed-off-by: Zhen Lei <thunder.leizhen@xxxxxxxxxx>
A nice improvement!
But wouldn't it be better to push this code down into dump_cpu_task()
itself, so that all callers could gain this benefit?
Thanx, Paul
> ---
> kernel/rcu/tree_stall.h | 18 +++++++++++++++++-
> 1 file changed, 17 insertions(+), 1 deletion(-)
>
> v1 --> v2:
> Fix C99 build warning:
> kernel/rcu/tree_stall.h:358:10: error: call to undeclared function 'get_irq_regs';
> ISO C99 and later do not support implicit function declarations
>
> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> index a001e1e7a99269c..23bfd755c3f6e5b 100644
> --- a/kernel/rcu/tree_stall.h
> +++ b/kernel/rcu/tree_stall.h
> @@ -8,6 +8,7 @@
> */
>
> #include <linux/kvm_para.h>
> +#include <asm/irq_regs.h>
>
> //////////////////////////////////////////////////////////////////////////////
> //
> @@ -350,6 +351,21 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> }
> #endif /* #else #ifdef CONFIG_PREEMPT_RCU */
>
> +static void rcu_dump_cpu_task(int cpu)
> +{
> + if (cpu == smp_processor_id() && in_irq()) {
> + struct pt_regs *regs;
> +
> + regs = get_irq_regs();
> + if (regs) {
> + show_regs(regs);
> + return;
> + }
> + }
> +
> + dump_cpu_task(cpu);
> +}
> +
> /*
> * Dump stacks of all tasks running on stalled CPUs. First try using
> * NMIs, but fall back to manual remote stack tracing on architectures
> @@ -369,7 +385,7 @@ static void rcu_dump_cpu_stacks(void)
> if (cpu_is_offline(cpu))
> pr_err("Offline CPU %d blocking current GP.\n", cpu);
> else if (!trigger_single_cpu_backtrace(cpu))
> - dump_cpu_task(cpu);
> + rcu_dump_cpu_task(cpu);
> }
> raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> }
> --
> 2.25.1
>
