Hi all,
On 6/28/24 1:15 오전, Sven Schnelle wrote:
> Yunseong Kim <yskelg@xxxxxxxxx> writes:
>
>> In my s390x archtecture, kprobe_eventname selftest have always failed
>> because of rcu_sched stalls.
>>
>> My environment is QEMU Ubuntu 24.04 KVM Machine Linux version
>> 6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13
>> (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36
>>
>> 1 configured CPUs, Linux is running under KVM in 64-bit mode
>>
>>
>> qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
>> -drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
>> -netdev user,id=enc0,hostfwd=tcp::10000-:22 \
>> -device virtio-net-ccw,netdev=enc0 \
>> -qmp tcp:localhost:4444,server,nowait
>>
>>
>> Currently, This failure can be always reproduced by this kselftests script:
>>
>> # tools/testing/selftests/ftrace/ftracetest
>> tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
> Could you see if you have the same issue on the most recent kernel?
>
> -- Steve
Thanks you Steve,
As soon as I saw your mail, I pulled the latest kernel.
Linux version 6.10.0-rc5+ (paran@fedora) (s390x-linux-gnu-gcc (GCC)
14.1.1 20240507 (Red Hat Cross 14.1.1-1), GNU ld version 2.41-1.fc40)
#30 SMP Fri Jun 28 04:32:20 KST 2024
My Linux is running under QEMU KVM in 64-bit mode, cpu: 1 configured
CPUs, 0 standby CPUs.
[ 726.069032] Kernel panic - not syncing: Corrupt kernel stack, can't
continue.
[ 726.069297] CPU: 0 PID: 1023 Comm: ftracetest Not tainted 6.10.0-rc5+ #30
[ 726.069335] Hardware name: QEMU 8561 QEMU (KVM/Linux)
[ 726.069361] Call Trace:
[ 726.069383] [<000003ffe0dc545e>] dump_stack_lvl+0x76/0xa8
[ 726.069446] [<000003ffe0db7170>] panic+0x170/0x358
[ 726.069485] [<000003ffe0000f16>] kernel_stack_overflow+0x56/0x60
[ 726.069522] [<0000000000000200>] 0x200
[ 726.069987] Dumping ftrace buffer:
[ 726.070144] ---------------------------------
[ 726.071356] <...>-1023 0..... 1829038us : p_kernel_clone_0:
(kernel_clone+0x0/0x390)
[ 726.071452] <...>-1023 0dN... 1829717us : p_kernel_clone_6:
(kernel_clone+0x6/0x390)
[ 726.071506] <...>-1023 0dN... 1830009us : p_kernel_clone_12:
(kernel_clone+0xc/0x390)
[ 726.071558] <...>-1023 0dN... 1830064us : p_kernel_clone_16:
...
I have changed the QEMU setting a bit to run latest kernel.
qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
-kernel bzImage-6.10.0-rc5-00035-gafcd48134c58 \
-initrd 6.8.0-36-generic/initrd.img-6.8.0-36-generic \
-append
"root=/dev/disk/by-id/dm-uuid-LVM-7JeH1Hjp3H1U60GH8m82Od6YXP47ItB01EcvyJ33Ya80VPeRinI1vfMjI9Ndzcc3
crashkernel=512M-:1024M oops=panic panic_on_warn=1
ftrace_dump_on_oops=orig_cpu slub_debug=UZ nokaslr gcov_persist=0 " \
-drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
-netdev user,id=enc0,hostfwd=tcp::10000-:22 \
-device virtio-net-ccw,netdev=enc0 \
-qmp tcp:localhost:4444,server,nowait
> Hi,
>
> Thanks for reporting!
Hi, Masami, It's my pleasure.
>> # tools/testing/selftests/ftrace/ftracetest
>> tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
>>
>> I've investigating cause of line, then I find this line.
>>
>> for i in `seq 0 255`; do
>> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
>> done
>>
>> cat kprobe_events >> $testlog
>>
>> echo 1 > events/kprobes/enable # <<<
>
> Isn't it kprobe_insn_boundary.tc?
That's exactly right!
I'm sorry, I slipped up again. :(
This is my exactly tested command.
$ cd tools/testing/selftests/ftrace/
$ sudo ./ftracetest --console test.d/kprobe/kprobe_insn_boundary.tc
> Also, can you modify it as
>
> for i in `seq 0 255`; do
> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
> echo 1 > events/kprobes/enable
> done
>
> And run it with --console option? It may tell us when the stall happens
> with what offset.
>
> Also, please try it with the latest kernel instead of 6.8.x
>
> Thank you,
>
I took what you said and applied it in test script.
--- a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
@@ -6,11 +6,11 @@
for i in `seq 0 255`; do
echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
+ echo 1 > events/kprobes/enable
done
cat kprobe_events >> $testlog
-echo 1 > events/kprobes/enable
( echo "forked" )
echo 0 > events/kprobes/enable
echo > kprobe_events
This is the results from console.
=== Ftrace unit tests ===
[1] Register multiple kprobe events in a function+ checkreq
/home/paran/linux-source/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
+ + grep ^#[ \t]*requires:
/home/paran/linux-source-6.8.0/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
cut -f2- -d:
+ requires= kprobe_events
+ eval check_requires kprobe_events
+ check_requires kprobe_events
+ p=kprobe_events
+ r=kprobe_events
+ t=kprobe_events
+ [ kprobe_events != kprobe_events ]
+ [ kprobe_events != kprobe_events ]
+ [ kprobe_events != kprobe_events ]
+ [ ! -e kprobe_events ]
+ initialize_ftrace
+ disable_tracing
+ echo 0
+ reset_tracer
+ echo nop
+ reset_trigger
+ [ -d events/synthetic ]
+ reset_trigger_file events/alarmtimer/alarmtimer_cancel/trigger
events/alarmtimer/alarmtimer_fired/trigger events/alarmtimer+
....
clear_trace
+ echo
+ enable_tracing
+ echo 1
+ .
/home/paran/linux-source/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
+ seq 0 255
+ echo p kernel_clone+0
+ echo 1
+ echo p kernel_clone+1
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+2
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+3
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+4
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+5
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+6
+ echo 1
+ echo p kernel_clone+7
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+8
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+9
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+10
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+11
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+12
+ echo 1
+ echo p kernel_clone+13
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+14
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+15
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+16
+ echo 1
+ echo p kernel_clone+17
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+18
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+19
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+20
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+21
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+22
+ echo 1
+ echo p kernel_clone+23
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+24
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+25
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+26
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+27
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+28
+ echo 1
+ echo p kernel_clone+29
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+30
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+31
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+32
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+33
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+34
+ echo 1
+ echo p kernel_clone+35
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+36
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+37
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+38
+ echo 1
+ echo p kernel_clone+39
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+40
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+41
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+42
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+43
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+44
+ echo 1
+ echo p kernel_clone+45
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+46
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+47
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+48
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+49
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+50
+ echo 1
+ echo p kernel_clone+51
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+52
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+53
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+54
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+55
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+56
+ echo 1
+ echo p kernel_clone+57
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+58
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+59
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+60
+ echo 1
+ echo p kernel_clone+61
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+62
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+63
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+64
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+65
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+66
+ echo 1
+ echo p kernel_clone+67
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+68
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+69
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+70
+ echo 1
+ echo p kernel_clone+71
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+72
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+73
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+74
+ echo 1
+ echo p kernel_clone+75
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+76
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+77
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+78
+ echo 1
+ echo p kernel_clone+79
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+80
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+81
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+82
+ echo 1
+ echo p kernel_clone+83
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+84
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+85
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+86
+ echo 1
+ echo p kernel_clone+87
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+88
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+89
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+90
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+91
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+92
+ echo 1
+ echo p kernel_clone+93
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+94
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+95
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+96
+ echo 1
+ echo p kernel_clone+97
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+98
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+99
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+100
+ echo 1
+ echo p kernel_clone+101
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+102
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+103
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+104
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+105
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+106
+ echo 1
+ echo p kernel_clone+107
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+108
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+109
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+110
+ echo 1
+ echo p kernel_clone+111
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+112
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+113
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+114
+ echo 1
+ echo p kernel_clone+115
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+116
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+117
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+118
+ echo 1
+ echo p kernel_clone+119
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+120
+ echo 1
+ echo p kernel_clone+121
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+122
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+123
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+124
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+125
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+126
+ echo 1
+ echo p kernel_clone+127
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+128
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+129
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+130
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+131
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+132
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+133
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+134
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+135
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+136
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+137
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+138
+ echo 1
+ echo p kernel_clone+139
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+140
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+141
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+142
+ echo 1
+ echo p kernel_clone+143
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+144
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+145
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+146
+ echo 1
+ echo p kernel_clone+147
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+148
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+149
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+150
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+151
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+152
+ echo 1
+ echo p kernel_clone+153
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+154
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+155
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+156
+ echo 1
+ echo p kernel_clone+157
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+158
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+159
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+160
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+161
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+162
+ echo 1
+ echo p kernel_clone+163
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+164
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+165
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+166
+ echo 1
+ echo p kernel_clone+167
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+168
+ echo 1
+ echo p kernel_clone+169
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+170
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+171
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+172
+ echo 1
+ echo p kernel_clone+173
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+174
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+175
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+176
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+177
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+178
+ echo 1
+ echo p kernel_clone+179
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+180
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+181
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+182
+ echo 1
+ echo p kernel_clone+183
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+184
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+185
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+186
+ echo 1
+ echo p kernel_clone+187
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+188
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+189
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+190
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+191
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+192
+ echo 1
+ echo p kernel_clone+193
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+194
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+195
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+196
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+197
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+198
+ echo 1
+ echo p kernel_clone+199
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+200
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+201
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+202
+ echo 1
+ echo p kernel_clone+203
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+204
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+205
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+206
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+207
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+208
+ echo 1
+ echo p kernel_clone+209
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+210
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+211
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+212
+ echo 1
+ echo p kernel_clone+213
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+214
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+215
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+216
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+217
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+218
+ echo 1
+ echo p kernel_clone+219
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+220
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+221
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+222
+ echo 1
+ echo p kernel_clone+223
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+224
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+225
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+226
+ echo 1
+ echo p kernel_clone+227
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+228
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+229
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+230
+ echo 1
+ echo p kernel_clone+231
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+232
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+233
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+234
+ echo 1
+ echo p kernel_clone+235
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+236
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+237
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+238
+ echo 1
+ echo p kernel_clone+239
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+240
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+241
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+242
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+243
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+244
+ echo 1
+ echo p kernel_clone+245
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+246
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+247
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+248
+ echo 1
+ echo p kernel_clone+249
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+250
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+251
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+252
>> I've investigating cause of line, then I find this line.
>>
>> for i in `seq 0 255`; do
>> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
>> done
>
> I think this is fixed with:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.10-rc5&id=c239c83ed5c558be3b5926c7f11639f02c8acd00
>
> Regards,
> Sven
Hi, Sven, Oh thank you so much for the link.
I have also checked that the patch(c239c83ed5c55) you mentioned has been
applied to the code.
It looks like have another problem.
This is the whole call trace.
[ 726.069032] Kernel panic - not syncing: Corrupt kernel stack, can't
continue.
[ 726.069297] CPU: 0 PID: 1023 Comm: ftracetest Not tainted 6.10.0-rc5+ #30
[ 726.069335] Hardware name: QEMU 8561 QEMU (KVM/Linux)
[ 726.069361] Call Trace:
[ 726.069383] [<000003ffe0dc545e>] dump_stack_lvl+0x76/0xa8
[ 726.069446] [<000003ffe0db7170>] panic+0x170/0x358
[ 726.069485] [<000003ffe0000f16>] kernel_stack_overflow+0x56/0x60
[ 726.069522] [<0000000000000200>] 0x200
[ 726.069987] Dumping ftrace buffer:
[ 726.070144] ---------------------------------
[ 726.071356] <...>-1023 0..... 1829038us : p_kernel_clone_0:
(kernel_clone+0x0/0x390)
[ 726.071452] <...>-1023 0dN... 1829717us : p_kernel_clone_6:
(kernel_clone+0x6/0x390)
[ 726.071506] <...>-1023 0dN... 1830009us : p_kernel_clone_12:
(kernel_clone+0xc/0x390)
[ 726.071558] <...>-1023 0dN... 1830064us : p_kernel_clone_16:
(kernel_clone+0x10/0x390)
[ 726.071610] <...>-1023 0dN... 1830116us : p_kernel_clone_22:
(kernel_clone+0x16/0x390)
[ 726.071661] <...>-1023 0dN... 1830161us : p_kernel_clone_28:
(kernel_clone+0x1c/0x390)
[ 726.071711] <...>-1023 0dN... 1830204us : p_kernel_clone_34:
(kernel_clone+0x22/0x390)
[ 726.071762] <...>-1023 0dN... 1830252us : p_kernel_clone_38:
(kernel_clone+0x26/0x390)
[ 726.071815] <...>-1023 0dN... 1830301us : p_kernel_clone_44:
(kernel_clone+0x2c/0x390)
[ 726.071866] <...>-1023 0dN... 1830355us : p_kernel_clone_50:
(kernel_clone+0x32/0x390)
[ 726.071917] <...>-1023 0dN... 1830399us : p_kernel_clone_56:
(kernel_clone+0x38/0x390)
[ 726.071967] <...>-1023 0dN... 1830441us : p_kernel_clone_60:
(kernel_clone+0x3c/0x390)
[ 726.072018] <...>-1023 0dN... 1830495us : p_kernel_clone_66:
(kernel_clone+0x42/0x390)
[ 726.072068] <...>-1023 0dN... 1830553us : p_kernel_clone_70:
(kernel_clone+0x46/0x390)
[ 726.072118] <...>-1023 0dN... 1830597us : p_kernel_clone_74:
(kernel_clone+0x4a/0x390)
[ 726.072168] <...>-1023 0dN... 1830639us : p_kernel_clone_78:
(kernel_clone+0x4e/0x390)
[ 726.072217] <...>-1023 0dN... 1830681us : p_kernel_clone_82:
(kernel_clone+0x52/0x390)
[ 726.072267] <...>-1023 0dN... 1830722us : p_kernel_clone_86:
(kernel_clone+0x56/0x390)
[ 726.072318] <...>-1023 0dN... 1830765us : p_kernel_clone_92:
(kernel_clone+0x5c/0x390)
[ 726.072368] <...>-1023 0dN... 1830806us : p_kernel_clone_96:
(kernel_clone+0x60/0x390)
[ 726.072418] <...>-1023 0dN... 1830847us : p_kernel_clone_100:
(kernel_clone+0x64/0x390)
[ 726.072468] <...>-1023 0dN... 1830911us : p_kernel_clone_106:
(kernel_clone+0x6a/0x390)
[ 726.072518] <...>-1023 0dN... 1830953us : p_kernel_clone_110:
(kernel_clone+0x6e/0x390)
[ 726.072568] <...>-1023 0dN... 1830994us : p_kernel_clone_114:
(kernel_clone+0x72/0x390)
[ 726.072619] <...>-1023 0dN... 1831035us : p_kernel_clone_118:
(kernel_clone+0x76/0x390)
[ 726.072669] <...>-1023 0dN... 1831092us : p_kernel_clone_120:
(kernel_clone+0x78/0x390)
[ 726.072718] <...>-1023 0dN... 1831133us : p_kernel_clone_126:
(kernel_clone+0x7e/0x390)
[ 726.072769] <...>-1023 0dN... 1831175us : p_kernel_clone_138:
(kernel_clone+0x8a/0x390)
[ 726.072819] <...>-1023 0dN... 1831221us : p_kernel_clone_142:
(kernel_clone+0x8e/0x390)
[ 726.072869] <...>-1023 0dN... 1831262us : p_kernel_clone_146:
(kernel_clone+0x92/0x390)
[ 726.072919] <...>-1023 0dN... 1831303us : p_kernel_clone_152:
(kernel_clone+0x98/0x390)
[ 726.072970] <...>-1023 0dN... 1831344us : p_kernel_clone_156:
(kernel_clone+0x9c/0x390)
[ 726.073020] <...>-1023 0dN... 1831385us : p_kernel_clone_162:
(kernel_clone+0xa2/0x390)
[ 726.073070] <...>-1023 0dN... 1831427us : p_kernel_clone_166:
(kernel_clone+0xa6/0x390)
[ 726.073120] <...>-1023 0dN... 1831469us : p_kernel_clone_168:
(kernel_clone+0xa8/0x390)
[ 726.073910] <...>-1023 0dN... 1831511us : p_kernel_clone_172:
(kernel_clone+0xac/0x390)
[ 726.073983] <...>-1023 0dN... 1831554us : p_kernel_clone_178:
(kernel_clone+0xb2/0x390)
[ 726.074047] <...>-1023 0dN... 1831601us : p_kernel_clone_182:
(kernel_clone+0xb6/0x390)
[ 726.074109] <...>-1023 0dN... 1831642us : p_kernel_clone_186:
(kernel_clone+0xba/0x390)
[ 726.074172] <...>-1023 0dN... 1831683us : p_kernel_clone_192:
(kernel_clone+0xc0/0x390)
[ 726.074234] <...>-1023 0dN... 1831725us : p_kernel_clone_198:
(kernel_clone+0xc6/0x390)
[ 726.074296] <...>-1023 0dN... 1831766us : p_kernel_clone_202:
(kernel_clone+0xca/0x390)
[ 726.074358] <...>-1023 0dN... 1831807us : p_kernel_clone_208:
(kernel_clone+0xd0/0x390)
[ 726.074419] <...>-1023 0dN... 1831848us : p_kernel_clone_212:
(kernel_clone+0xd4/0x390)
[ 726.074481] <...>-1023 0dN... 1831895us : p_kernel_clone_218:
(kernel_clone+0xda/0x390)
[ 726.074542] <...>-1023 0dN... 1831941us : p_kernel_clone_222:
(kernel_clone+0xde/0x390)
[ 726.074603] <...>-1023 0dN... 1831982us : p_kernel_clone_226:
(kernel_clone+0xe2/0x390)
[ 726.074665] <...>-1023 0dN... 1832023us : p_kernel_clone_230:
(kernel_clone+0xe6/0x390)
[ 726.074727] <...>-1023 0dN... 1832064us : p_kernel_clone_234:
(kernel_clone+0xea/0x390)
[ 726.074787] <...>-1023 0dN... 1832105us : p_kernel_clone_238:
(kernel_clone+0xee/0x390)
[ 726.074846] <...>-1023 0dN... 1832146us : p_kernel_clone_244:
(kernel_clone+0xf4/0x390)
[ 726.074906] <...>-1023 0dN... 1832188us : p_kernel_clone_248:
(kernel_clone+0xf8/0x390)
[ 726.074965] <...>-1023 0dN... 1832229us : p_kernel_clone_252:
(kernel_clone+0xfc/0x390)
[ 726.075024] <...>-1023 0dN... 1832286us : p_kernel_clone_74:
(kernel_clone+0x4a/0x390)
[ 726.075083] <...>-1023 0dN... 1832324us : p_kernel_clone_78:
(kernel_clone+0x4e/0x390)
[ 726.075142] <...>-1023 0dN... 1832361us : p_kernel_clone_82:
(kernel_clone+0x52/0x390)
[ 726.075200] <...>-1023 0dN... 1832398us : p_kernel_clone_86:
(kernel_clone+0x56/0x390)
[ 726.075259] <...>-1023 0dN... 1832436us : p_kernel_clone_92:
(kernel_clone+0x5c/0x390)
[ 726.075317] <...>-1023 0dN... 1832472us : p_kernel_clone_96:
(kernel_clone+0x60/0x390)
[ 726.075376] <...>-1023 0dN... 1832510us : p_kernel_clone_100:
(kernel_clone+0x64/0x390)
[ 726.075434] <...>-1023 0dN... 1832547us : p_kernel_clone_106:
(kernel_clone+0x6a/0x390)
[ 726.075493] <...>-1023 0dN... 1832584us : p_kernel_clone_110:
(kernel_clone+0x6e/0x390)
[ 726.075552] <...>-1023 0dN... 1832620us : p_kernel_clone_114:
(kernel_clone+0x72/0x390)
[ 726.075611] <...>-1023 0dN... 1832657us : p_kernel_clone_118:
(kernel_clone+0x76/0x390)
[ 726.075669] <...>-1023 0dN... 1832694us : p_kernel_clone_120:
(kernel_clone+0x78/0x390)
[ 726.075734] <...>-1023 0dN... 1832731us : p_kernel_clone_126:
(kernel_clone+0x7e/0x390)
[ 726.075793] <...>-1023 0dN... 1832768us : p_kernel_clone_138:
(kernel_clone+0x8a/0x390)
[ 726.075860] <...>-1023 0dN... 1832805us : p_kernel_clone_142:
(kernel_clone+0x8e/0x390)
[ 726.075922] <...>-1023 0dN... 1832848us : p_kernel_clone_146:
(kernel_clone+0x92/0x390)
[ 726.075980] <...>-1023 0dN... 1832885us : p_kernel_clone_152:
(kernel_clone+0x98/0x390)
[ 726.076046] <...>-1023 0dN... 1832922us : p_kernel_clone_156:
(kernel_clone+0x9c/0x390)
[ 726.076107] <...>-1023 0dN... 1832959us : p_kernel_clone_162:
(kernel_clone+0xa2/0x390)
[ 726.076168] <...>-1023 0dN... 1832996us : p_kernel_clone_166:
(kernel_clone+0xa6/0x390)
[ 726.076229] <...>-1023 0dN... 1833038us : p_kernel_clone_168:
(kernel_clone+0xa8/0x390)
[ 726.076290] <...>-1023 0dN... 1833078us : p_kernel_clone_172:
(kernel_clone+0xac/0x390)
[ 726.076376] <...>-1023 0dN... 1833117us : p_kernel_clone_178:
(kernel_clone+0xb2/0x390)
[ 726.076438] <...>-1023 0dN... 1833154us : p_kernel_clone_182:
(kernel_clone+0xb6/0x390)
[ 726.076505] <...>-1023 0dN... 1833190us : p_kernel_clone_186:
(kernel_clone+0xba/0x390)
[ 726.076565] <...>-1023 0dN... 1833229us : p_kernel_clone_192:
(kernel_clone+0xc0/0x390)
[ 726.076634] <...>-1023 0dN... 1833266us : p_kernel_clone_198:
(kernel_clone+0xc6/0x390)
[ 726.076695] <...>-1023 0dN... 1833302us : p_kernel_clone_202:
(kernel_clone+0xca/0x390)
[ 726.076779] <...>-1023 0dN... 1833339us : p_kernel_clone_208:
(kernel_clone+0xd0/0x390)
[ 726.076842] <...>-1023 0dN... 1833376us : p_kernel_clone_212:
(kernel_clone+0xd4/0x390)
[ 726.076886] ---------------------------------
Thank you Steve, Masami and Sven for the great advice on what I reported.
If there are any additional aspects I should check or any advice,
please let me know and I will look into it.
Warm regards,
Yunseong Kim
