Hi all,
In my s390x archtecture, kprobe_eventname selftest have always failed
because of rcu_sched stalls.
My environment is QEMU Ubuntu 24.04 KVM Machine Linux version
6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13
(Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36
1 configured CPUs, Linux is running under KVM in 64-bit mode
qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
-drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
-netdev user,id=enc0,hostfwd=tcp::10000-:22 \
-device virtio-net-ccw,netdev=enc0 \
-qmp tcp:localhost:4444,server,nowait
Currently, This failure can be always reproduced by this kselftests script:
# tools/testing/selftests/ftrace/ftracetest
tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
I've investigating cause of line, then I find this line.
for i in `seq 0 255`; do
echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
done
cat kprobe_events >> $testlog
echo 1 > events/kprobes/enable # <<<
This line makes "rcu_sched detected stalls" log and stall the system.
[ 7825.578940] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
After this line, the test doesn't go any further.
This test was added in the patch below.
("selftests/ftrace: Add new test case which adds multiple consecutive
probes in a function")
Link:
https://lore.kernel.org/linux-trace-kernel/20230428163842.95118-2-akanksha@xxxxxxxxxxxxx/
I've attached a link to a compressed version of vmcore, vmlinux and
config files that I dumped from my environment.
https://drive.google.com/file/d/1O2bCKrRbyJ-yP4zTz_sAd_qM80nHnCGr/view?usp=sharing
I used QEMU QMP to dump the vmcore.
$ telnet localhost 4444
{"execute": "qmp_capabilities"}
{"execute":"dump-guest-memory","arguments":
{"paging":false,"protocol":"file:/home/paran/vmcore1.img"}}
rcu: INFO: rcu_sched detected stalls on CPUs/tasks:s:
rcu:
(detected by 0, t=6002 jiffies, g=24353, q=1 ncpus=1)1)
rcu: All QSes seen, last rcu_sched kthread activity 6002
(4294978930-4294972928), jiffies_till_next_fqs=1, root ->qsmask 0x0x0
rcu: rcu_sched kthread starved for 6002 jiffies! g24353 f0x2
RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0=0
rcu:
Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected
behavior.r.
rcu: RCU grace-period kthread stack dump:p:
task:rcu_sched state:R running task stack:0 pid:16
tgid:16 ppid:2 flags:0x0000000000
Call Trace:e:
__schedule+0x346/0x8b8 8
schedule+0x36/0x148 8
schedule_timeout+0x8e/0x148 8
rcu_gp_fqs_loop+0x444/0x548 8
rcu_gp_kthread+0x146/0x198 8
kthread+0x124/0x128 8
__ret_from_fork+0x40/0x58 8
ret_from_fork+0xa/0x30 0
rcu: Stack dump where RCU GP kthread last ran:n:
CPU: 0 PID: 1077 Comm: ftracetest Not tainted 6.8.0-36-generic #36-Ubuntu
Hardware name: QEMU 8561 QEMU (KVM/Linux)
Krnl PSW : 0704f00180000000 0000000000121d32 kprobe_exceptions_notify
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:519
(discriminator 1))
R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 RI:0 EA:3:3
Krnl GPRS: 0000000000000000 0000000000000000 0000000000008001
0000000000000000
0000000000000000 0000000000000000 0000000000000000
0000000001956720
ffffffffffffffff 0000000000121c98 0000000001958d40
00000380000cfbd8
000003ff938dbc78 00000380000cfab8 0000000000121d1c
00000380000cf980
Krnl Code: 0000000000121d26: 9103b008
Code starting with the faulting instruction
===========================================
8(%r11),3
0000000000121d2a: a7840004
8,0000000000121d32
#0000000000121d2e: ad03f0a0
stosm
160(%r15),3
>0000000000121d32: b9140022
lgfr
%r2,%r2
0000000000121d36: ebbff0a80004
%r11,%r15,168(%r15)
0000000000121d3c: a7190000
lghi
%r1,0
0000000000121d40: a7390000
lghi
%r3,0
0000000000121d44: a7490000
lghi
%r4,0
Call Trace:
kprobe_exceptions_notify
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:519
(discriminator 1))
kprobe_exceptions_notify
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:504
(discriminator 1))
notifier_call_chain (/build/linux-3nCxw2/linux-6.8.0/kernel/notifier.c:93)
notify_die (/build/linux-3nCxw2/linux-6.8.0/kernel/notifier.c:597)
do_per_trap
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:75
(discriminator 1))
__do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/irqflags.h:47
(discriminator 1)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/irqflags.h:52
(discriminator 1)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:356
(discriminator 1))
pgm_check_handler
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:383)
kernel_clone
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/uaccess.h:119
(discriminator 1) /build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:2927
(discriminator 1))
__do_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3055)
__s390x_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3027)
__do_syscall
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:195
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:200
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:145
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:168
(discriminator 3))
system_call (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:309)
Last Breaking-Event-Address:
0xfdf5045050
?rcu: INFO: rcu_sched detected stalls on CPUs/tasks:s:
@rcu:
(detected by 0, t=24007 jiffies, g=24353, q=1 ncpus=1)
rcu: All QSes seen, last rcu_sched kthread activity 24007
(4294996935-4294972928), jiffies_till_next_fqs=1, root ->qsmask 0x0x0
rcu: rcu_sched kthread starved for 24007 jiffies! g24353 f0x2
RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0=0
rcu:
Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected
behavior.r.
rcu: RCU grace-period kthread stack dump:p:
task:rcu_sched state:R running task stack:0 pid:16
tgid:16 ppid:2 flags:0x0000000000
Call Trace:e:
__schedule+0x346/0x8b8 8
schedule+0x36/0x148 8
schedule_timeout+0x8e/0x148 8
rcu_gp_fqs_loop+0x444/0x548 8
rcu_gp_kthread+0x146/0x198 8
kthread+0x124/0x128 8
__ret_from_fork+0x40/0x58 8
ret_from_fork+0xa/0x30 0
rcu: Stack dump where RCU GP kthread last ran:n:
CPU: 0 PID: 1077 Comm: ftracetest Not tainted 6.8.0-36-generic #36-Ubuntu
Hardware name: QEMU 8561 QEMU (KVM/Linux)
Krnl PSW : 0704d00180000000 0000000000ebe0b2 __do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:353)
R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3:3
Krnl GPRS: 0704c00180000000 0000000000000000 00000380000cfb97
0000000000000000
0000000000000000 0704c00180000000 0000000000000000
0000000000000000
0704c00180000000 00000000001a8388 0000000000000000
00000380000cfbd8
000003ff938dbc78 0000000000ed1c6c 0000000000ebe024
00000380000cfaf0
Krnl Code: 0000000000ebe0a4: a504bfff
Code starting with the faulting instruction
===========================================
nihh
%r0,49151
0000000000ebe0a8: e300f0a80024
%r0,168(%r15)
#0000000000ebe0ae: 8000f0a8
168(%r15)
>0000000000ebe0b2: 5850b0a0
%r5,160(%r11)
0000000000ebe0b6: c05b0000007f
nilf
%r5,127
0000000000ebe0bc: a7840012
8,0000000000ebe0e0
0000000000ebe0c0: b91600e5
llgfr
%r14,%r5
0000000000ebe0c4: c0400051121e
larl
%r4,00000000018e0500
Call Trace:
__do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:353)
__do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:318)
pgm_check_handler
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:383)
kernel_clone
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/uaccess.h:119
(discriminator 1) /build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:2927
(discriminator 1))
__do_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3055)
__s390x_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3027)
__do_syscall
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:195
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:200
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:145
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:168
(discriminator 3))
system_call (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:309)
Last Breaking-Event-Address:
0x4404c0018000000000
It's not easy for me to resolve this issue. If advice or guidance can be
provided on how to resolve this issue, I'll try sending a patch!
Warm regards,
Yunseong Kim
