On Wed, Feb 22, 2023 at 08:40:52PM +0800, Yacan Liu wrote: > Hi folks: > > We recently encountered a kernel stall issue related to infiniband mad > packet. We believe this is caused by some abnormal operations of user-level > programs.But I am not sure if the kernel driver needs to do some protection. > > In our cluster, multiple nodes will establish multiple RDMA connections with > each other. Then, when we start the program, the kernel prompts the > following error after one minute: > > [ 5489.530125] rcu: INFO: rcu_sched self-detected stall on CPU > [ 5489.535714] rcu: 55-....: (19027 ticks this GP) > idle=236/1/0x4000000000000000 softirq=13510/13510 fqs=4359 > [ 5489.545554] (t=21015 jiffies g=279845 q=701559) > [ 5489.545557] NMI backtrace for cpu 55 > [ 5489.545559] CPU: 55 PID: 44979 Comm: kworker/u196:1 Tainted: G OE > 5.10.91-netease3-1 #2 > [ 5489.545560] Hardware name: Inspur SA5212M5/YZMB-00882-10T, BIOS 4.1.21 > 08/25/2021 > [ 5489.545582] Workqueue: ib-comp-unb-wq ib_cq_poll_work [ib_core] > [ 5489.545584] Call Trace: > [ 5489.545586] > [ 5489.545590] dump_stack+0x6b/0x83 > [ 5489.545593] nmi_cpu_backtrace.cold+0x32/0x69 > [ 5489.545596] ? lapic_can_unplug_cpu+0x80/0x80 > [ 5489.545599] nmi_trigger_cpumask_backtrace+0xd7/0xe0 > [ 5489.545602] rcu_dump_cpu_stacks+0xa2/0xd0 > [ 5489.545604] rcu_sched_clock_irq.cold+0x1ff/0x3d6 > [ 5489.545607] ? trigger_load_balance+0x5a/0x240 > [ 5489.545610] update_process_times+0x8c/0xc0 > [ 5489.545613] tick_sched_handle+0x22/0x60 > [ 5489.545615] tick_sched_timer+0x7c/0xb0 > [ 5489.545617] ? tick_do_update_jiffies64.part.0+0xc0/0xc0 > [ 5489.545619] __hrtimer_run_queues+0x12a/0x270 > [ 5489.545621] hrtimer_interrupt+0x110/0x2c0 > [ 5489.545623] __sysvec_apic_timer_interrupt+0x5f/0xd0 > [ 5489.545627] asm_call_irq_on_stack+0xf/0x20 > [ 5489.545627] > [ 5489.545629] sysvec_apic_timer_interrupt+0x72/0x80 > [ 5489.545631] asm_sysvec_apic_timer_interrupt+0x12/0x20 > > OS: Debian 11 > Kernel: 5.10.91 > ofed: MLNX_OFED_LINUX-5.5-1.0.3.2 Please approach your NVidia support representative. There are some differences between upstream and MOFED code in MAD area. Thanks
