Hi, Aditya
Thank you for the update.
On 12/20/23 16:52, Aditya Gupta wrote:
The Problem:
============
Currently crash is unable to show function arguments and local variables, as
gdb can do. And functionality for moving between frames ('up'/'down') is not
working in crash.
Crash has 'gdb passthroughs' for things gdb can do, but the gdb passthroughs
'bt', 'frame', 'info locals', 'up', 'down' are not working either, due to
gdb not getting the register values from `crash_target::fetch_registers`,
which then uses `machdep->get_cpu_reg`, which is not implemented for PPC64
Proposed Solution:
==================
Fix the gdb passthroughs by implementing "machdep->get_cpu_reg" for PPC64.
This way, "gdb mode in crash" will support this feature for both ELF and
kdump-compressed vmcore formats, while "gdb" would only have supported ELF
format
This way other features of 'gdb', such as seeing
backtraces/registers/variables/arguments/local variables, moving up and
down stack frames, can be used with any ppc64 vmcore, irrespective of
being ELF format or kdump-compressed format.
Note: This doesn't support live debugging on ppc64, since registers are not
available to be read
Implications on Architectures:
====================================
No architecture other than PPC64 has been affected, other than in case of
'frame' command
As mentioned in patch #2, since frame will not be prohibited, so it will print:
crash> frame
#0 <unavailable> in ?? ()
Instead of before prohibited message:
crash> frame
crash: prohibited gdb command: frame
Major change will be in 'gdb mode' on PPC64, that it will print the frames, and
local variables, instead of failing with errors showing no frame, or showing
that couldn't get PC, it will be able to give all this information.
Testing:
========
Git tree with this patch series applied:
https://github.com/adi-g15-ibm/crash/tree/stack-unwind-v5-smaller-gran
To test various gdb passthroughs:
(crash) set
(crash) set gdb on
gdb> thread
gdb> bt
gdb> info threads
gdb> info threads
gdb> info locals
gdb> info variables irq_rover_lock
gdb> info args
gdb> thread 2
gdb> set gdb off
(crash) set
(crash) set -c 6
(crash) gdb thread
(crash) bt
(crash) gdb bt
(crash) frame
(crash) up
(crash) down
(crash) info locals
After applying the v5, I still got the following errors:
crash> gdb bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
crash> gdb frame
#0 <unavailable> in ?? ()
crash> set gdb on
gdb: on
gdb> bt
#0 <unavailable> in ?? ()
#1 0xc0000000000f570c in plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:111
#2 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#3 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#4 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#5 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#6 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#7 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#8 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#9 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#11 0xc000000000011030 in rest_init () at init/main.c:730
#12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
#13 0xc0000000020051ec in start_kernel () at init/main.c:1072
#14 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
gdb> info threads
Id Target Id Frame
* 1 CPU 0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
2 CPU 1 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
3 CPU 2 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
4 CPU 3 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
5 CPU 4 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
6 CPU 5 0xc00000000028b5e8 in crash_setup_regs
(oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
./arch/powerpc/include/asm/kexec.h:69
7 CPU 6 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
8 CPU 7 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem_callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem_callback"
gdb> thread 1
[Switching to thread 1 (CPU 0)]
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
114 li r4,0
gdb> bt
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
#1 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#4 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#10 0xc000000000011030 in rest_init () at init/main.c:730
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c80 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c7c type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c000000002004c78 type:
"gdb_readmem callback"
#11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051ec type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e8 type:
"gdb_readmem callback"
gdb: page excluded: kernel virtual address: c0000000020051e4 type:
"gdb_readmem callback"
#12 0xc0000000020051ec in start_kernel () at init/main.c:1072
#13 0xc00000000000e788 in start_here_common () at
arch/powerpc/kernel/head_64.S:1039
gdb> thread 3
[Switching to thread 3 (CPU 2)]
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
114 li r4,0
gdb> bt
#0 plpar_hcall_norets_notrace () at
arch/powerpc/platforms/pseries/hvCall.S:114
#1 0xc000000001004dd8 in cede_processor () at
./arch/powerpc/include/asm/plpar_wrappers.h:37
#2 check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
#3 0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
drv=<optimized out>, index=<optimized out>) at
drivers/cpuidle/cpuidle-pseries.c:256
#4 0xc000000001004498 in cpuidle_enter_state
(dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
<pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
#5 0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
<pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized out>) at
drivers/cpuidle/cpuidle.c:388
#6 0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
kernel/sched/idle.c:134
#7 0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
#8 0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
#9 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
kernel/sched/idle.c:380
#10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at
arch/powerpc/kernel/smp.c:1680
#11 0xc00000000000e058 in start_secondary_prolog () at
arch/powerpc/kernel/head_64.S:885
gdb>
Could you please check it again? Or am I missing anything? I did the
test based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
BTW: I did not see the similar issues after applying the v4 patch set.
Thanks
Lianbo
Known Issues:
=============
1. In gdb mode, 'bt' might fail to show backtrace in few vmcores collected
from older kernels. This is a known issue due to register mismatch, and
its fix has been merged upstream:
This can also cause some 'invalid kernel virtual address' errors during gdb
unwinding the stack registers
Commit: https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef785819e72db79
Fixing GDB passthroughs on other architectures
==============================================
Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
thread', 'gdb info locals' etc. has been done by the patches introducing
'machdep->get_cpu_reg' and this series fixing some issues in that.
Other architectures should be able to fix these gdb functionalities by
simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
The reasoning behind that has been explained with a diagram in commit
description of patch #1
I will assist with my findings/observations fixing it on ppc64 whenever needed.
Changelog:
==========
V5:
+ changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
code
+ changes in patch #3: fixed typo 'ppc64_renum' instead of 'ppc64_regnum',
remove unneeded if condition
+ changes in patch #5: implement refresh regcache on per thread, instead of all
threads at once
V4:
+ fix segmentation fault in live debugging (change in patch #1)
+ mention live debugging not supported in cover letter and patch #1
+ fixed some checkpatch warnings (change in patch #5)
V3:
+ default gdb thread will be the crashing thread, instead of being
thread '0'
+ synchronise crash cpu and gdb thread context
+ fix bug in gdb_interface, that replaced gdb's output stream, losing
output in some cases, such as info threads and extra output in info
variables
+ fix 'info threads'
RFC V2:
- removed patch implementing 'frame', 'up', 'down' in crash
- updated the cover letter by removing the mention of those commands other
than the respective gdb passthrough
Aditya Gupta (5):
ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
remove 'frame' from prohibited commands list
synchronise cpu context changes between crash/gdb
fix gdb_interface: restore gdb's output streams at end of
gdb_interface
fix 'info threads' command
crash_target.c | 44 ++++++++++++++++
defs.h | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
gdb-10.2.patch | 110 +++++++++++++++++++++++++++++++++++++++-
gdb_interface.c | 2 +-
kernel.c | 47 +++++++++++++++--
ppc64.c | 95 +++++++++++++++++++++++++++++++++--
task.c | 14 ++++++
tools.c | 2 +-
8 files changed, 434 insertions(+), 10 deletions(-)
--
Crash-utility mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxxxxxx
https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
Contribution Guidelines: https://github.com/crash-utility/crash/wiki
