[Crash-utility] Re: [PATCH v5 0/5] Improve stack unwind on ppc64

[Lianbo Jiang <lijiang@xxxxxxxxxx>]

On 1/5/24 15:07, Aditya Gupta wrote:

> Hi Lianbo,
>
> On Fri, Jan 05, 2024 at 01:53:53PM +0800, Lianbo Jiang wrote:
> > > ...<snip>...
> > >
> > > Q. Is it harmless ?
> > >
> > > Yes, all unwinders which were tried before chosing the correct one, are expected
> > > to fail.
> > >
> > > About address being unavailable, that also is harmless because:
> > > 1. The functions instructions were freed by the kernel, so the data at those
> > > address being missing is expected
> > > 2. Even GDB is okay with a read failing, in this case, tramp_frame_sniffer
> > > simply returns if it's unable to get an address.
> > >
> > > ------------
> > >
> > > Btw, this only happens with CPU0 WHEN it has idle task. Since unlike other CPUs
> > > idle tasks, CPU 0's idle task is called during this initialisation sequence,
> > > thus it's backtrace having very early init functions.
> > It's true.
> > > Since this is expected and harmless (gdb handles such cases), I am thinking to
> > > ignore this warning when called by gdb, what do you suggest ?
> > If this does happen and it's rare, it might be good to have a warning for
> > users.
> >
> > At least it reflects the actual situation, although the warning output looks
> > not very friendly. So I would suggest still leaving it there for the time
> > being.
> Yes, the warning is not very friendly but it does tell that the page is
> excluded.
>
> > >       i) Add flag "IGNORE_PAGE_EXCLUDED" to 'readmem' call when done by gdb, and
> > >          only print the error if this is true:
> > >       	if ((error_handle & IGNORE_PAGE_EXCLUDED) && (CRASHDEBUG(0))
> > >       		...
> > >       This way, it will still be visible with -d 1 and above debug levels.
> > >       ii) Append 'quiet' to the 'error_handle':
> > >       This way we don't have to introduce a flag, but it might skip showing other
> > >       warnings/errors which might be of interest
> > >
> > > Other than this, there was a bug introduced in V5, where default thread is still
> > > thread 1 (CPU 0) at initialisation, which has been fixed, and will also fix the
> > Sounds good.
> So for now there's no change needed to the 'page excluded' warnings ?

Yes, no need changes this time.

I did not see the similar warnings in other situations, unless there are 
a lot of warnings for this cases.

Thanks

Lianbo

> > > 'unavailable' frame being shown. Though it's funny that bug helped in a way to
> > > uncover these warnings, this would have hit in earlier versions also.
> > Indeed, it can be triggered conditionally.
> >
> > I have no other issues, thank you for the great job, Aditya.
> Thanks,
> Aditya Gupta
>
> > Thanks
> >
> > Lianbo
> >
> > > Thanks,
> > > Aditya Gupta
> > >
> > > > crash> gdb frame
> > > > #0  <unavailable> in ?? ()
> > > > crash> set gdb on
> > > > gdb: on
> > > > gdb> bt
> > > > #0  <unavailable> in ?? ()
> > > > #1  0xc0000000000f570c in plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:111
> > > > #2  0xc000000001004dd8 in cede_processor () at
> > > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > > #3  check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> > > > #4  0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> > > > drv=<optimized out>, index=<optimized out>) at
> > > > drivers/cpuidle/cpuidle-pseries.c:256
> > > > #5  0xc000000001004498 in cpuidle_enter_state
> > > > (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> > > > <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> > > > #6  0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > > <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
> > > > drivers/cpuidle/cpuidle.c:388
> > > > #7  0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> > > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> > > > dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> > > > kernel/sched/idle.c:134
> > > > #8  0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> > > > #9  0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > > #10 0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> > > > kernel/sched/idle.c:380
> > > > #11 0xc000000000011030 in rest_init () at init/main.c:730
> > > > #12 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> > > > #13 0xc0000000020051ec in start_kernel () at init/main.c:1072
> > > > #14 0xc00000000000e788 in start_here_common () at
> > > > arch/powerpc/kernel/head_64.S:1039
> > > > gdb> info threads
> > > >     Id   Target Id         Frame
> > > > * 1    CPU 0             plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > >     2    CPU 1             plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > >     3    CPU 2             plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > >     4    CPU 3             plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > >     5    CPU 4             plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > >     6    CPU 5             0xc00000000028b5e8 in crash_setup_regs
> > > > (oldregs=<optimized out>, newregs=0xc00000005d3e7958) at
> > > > ./arch/powerpc/include/asm/kexec.h:69
> > > >     7    CPU 6             plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > >     8    CPU 7             plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > > "gdb_readmem_callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > > "gdb_readmem_callback"
> > > >
> > > > gdb> thread 1
> > > > [Switching to thread 1 (CPU 0)]
> > > > #0  plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > 114             li      r4,0
> > > > gdb> bt
> > > > #0  plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > #1  0xc000000001004dd8 in cede_processor () at
> > > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > > #2  check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> > > > #3  0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> > > > drv=<optimized out>, index=<optimized out>) at
> > > > drivers/cpuidle/cpuidle-pseries.c:256
> > > > #4  0xc000000001004498 in cpuidle_enter_state
> > > > (dev=dev@entry=0xc0000001ff5910c0, drv=drv@entry=0xc000000002b8f558
> > > > <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> > > > #5  0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > > <pseries_idle_driver>, dev=0xc0000001ff5910c0, index=<optimized out>) at
> > > > drivers/cpuidle/cpuidle.c:388
> > > > #6  0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> > > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> > > > dev@entry=0xc0000001ff5910c0, next_state=<optimized out>) at
> > > > kernel/sched/idle.c:134
> > > > #7  0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> > > > #8  0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > > #9  0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> > > > kernel/sched/idle.c:380
> > > > #10 0xc000000000011030 in rest_init () at init/main.c:730
> > > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c80 type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c7c type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c000000002004c78 type:
> > > > "gdb_readmem callback"
> > > > #11 0xc000000002004c80 in arch_call_rest_init () at init/main.c:827
> > > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051ec type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e8 type:
> > > > "gdb_readmem callback"
> > > > gdb: page excluded: kernel virtual address: c0000000020051e4 type:
> > > > "gdb_readmem callback"
> > > > #12 0xc0000000020051ec in start_kernel () at init/main.c:1072
> > > > #13 0xc00000000000e788 in start_here_common () at
> > > > arch/powerpc/kernel/head_64.S:1039
> > > > gdb> thread 3
> > > > [Switching to thread 3 (CPU 2)]
> > > > #0  plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > 114             li      r4,0
> > > > gdb> bt
> > > > #0  plpar_hcall_norets_notrace () at
> > > > arch/powerpc/platforms/pseries/hvCall.S:114
> > > > #1  0xc000000001004dd8 in cede_processor () at
> > > > ./arch/powerpc/include/asm/plpar_wrappers.h:37
> > > > #2  check_and_cede_processor () at drivers/cpuidle/cpuidle-pseries.c:83
> > > > #3  0xc000000001005000 in shared_cede_loop (dev=<optimized out>,
> > > > drv=<optimized out>, index=<optimized out>) at
> > > > drivers/cpuidle/cpuidle-pseries.c:256
> > > > #4  0xc000000001004498 in cpuidle_enter_state
> > > > (dev=dev@entry=0xc0000001ff7910c0, drv=drv@entry=0xc000000002b8f558
> > > > <pseries_idle_driver>, index=index@entry=1) at drivers/cpuidle/cpuidle.c:267
> > > > #5  0xc000000000c0eb4c in cpuidle_enter (drv=0xc000000002b8f558
> > > > <pseries_idle_driver>, dev=0xc0000001ff7910c0, index=<optimized out>) at
> > > > drivers/cpuidle/cpuidle.c:388
> > > > #6  0xc0000000001ce2bc in call_cpuidle (drv=<optimized out>,
> > > > drv@entry=0xc000000002b8f558 <pseries_idle_driver>, dev=<optimized out>,
> > > > dev@entry=0xc0000001ff7910c0, next_state=<optimized out>) at
> > > > kernel/sched/idle.c:134
> > > > #7  0xc0000000001d5d68 in cpuidle_idle_call () at kernel/sched/idle.c:215
> > > > #8  0xc0000000001d5f58 in do_idle () at kernel/sched/idle.c:282
> > > > #9  0xc0000000001d6298 in cpu_startup_entry (state=<optimized out>) at
> > > > kernel/sched/idle.c:380
> > > > #10 0xc00000000005f048 in start_secondary (unused=<optimized out>) at
> > > > arch/powerpc/kernel/smp.c:1680
> > > > #11 0xc00000000000e058 in start_secondary_prolog () at
> > > > arch/powerpc/kernel/head_64.S:885
> > > > gdb>
> > > >
> > > >
> > > > Could you please check it again? Or am I missing anything? I did the test
> > > > based on upstream kernel 6.7.0-rc7(commit:8735c7c84d1b).
> > > >
> > > > BTW: I did not see the similar issues after applying the v4 patch set.
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Lianbo
> > > >
> > > > > Known Issues:
> > > > > =============
> > > > >
> > > > > 1. In gdb mode, 'bt' might fail to show backtrace in few vmcores collected
> > > > >       from older kernels. This is a known issue due to register mismatch, and
> > > > >       its fix has been merged upstream:
> > > > >
> > > > >       This can also cause some 'invalid kernel virtual address' errors during gdb
> > > > >       unwinding the stack registers
> > > > >
> > > > > Commit: https://github.com/torvalds/linux/commit/b684c09f09e7a6af3794d4233ef785819e72db79
> > > > >
> > > > > Fixing GDB passthroughs on other architectures
> > > > > ==============================================
> > > > >
> > > > > Much of the work for making gdb passthroughs like 'gdb bt', 'gdb
> > > > > thread', 'gdb info locals' etc. has been done by the patches introducing
> > > > > 'machdep->get_cpu_reg' and this series fixing some issues in that.
> > > > >
> > > > > Other architectures should be able to fix these gdb functionalities by
> > > > > simply implementing 'machdep->get_cpu_reg (cpu, regno, ...)'.
> > > > >
> > > > > The reasoning behind that has been explained with a diagram in commit
> > > > > description of patch #1
> > > > >
> > > > > I will assist with my findings/observations fixing it on ppc64 whenever needed.
> > > > >
> > > > > Changelog:
> > > > > ==========
> > > > >
> > > > > V5:
> > > > > + changes in patch #1: made ppc64_get_cpu_reg static, and remove unreachable
> > > > > code
> > > > > + changes in patch #3: fixed typo 'ppc64_renum' instead of 'ppc64_regnum',
> > > > > remove unneeded if condition
> > > > > + changes in patch #5: implement refresh regcache on per thread, instead of all
> > > > > threads at once
> > > > >
> > > > > V4:
> > > > > + fix segmentation fault in live debugging (change in patch #1)
> > > > > + mention live debugging not supported in cover letter and patch #1
> > > > > + fixed some checkpatch warnings (change in patch #5)
> > > > >
> > > > > V3:
> > > > > + default gdb thread will be the crashing thread, instead of being
> > > > >      thread '0'
> > > > > + synchronise crash cpu and gdb thread context
> > > > > + fix bug in gdb_interface, that replaced gdb's output stream, losing
> > > > >      output in some cases, such as info threads and extra output in info
> > > > >      variables
> > > > > + fix 'info threads'
> > > > >
> > > > > RFC V2:
> > > > >      - removed patch implementing 'frame', 'up', 'down' in crash
> > > > >      - updated the cover letter by removing the mention of those commands other
> > > > > 	than the respective gdb passthrough
> > > > >
> > > > > Aditya Gupta (5):
> > > > >      ppc64: correct gdb passthroughs by implementing machdep->get_cpu_reg
> > > > >      remove 'frame' from prohibited commands list
> > > > >      synchronise cpu context changes between crash/gdb
> > > > >      fix gdb_interface: restore gdb's output streams at end of
> > > > >        gdb_interface
> > > > >      fix 'info threads' command
> > > > >
> > > > >     crash_target.c  |  44 ++++++++++++++++
> > > > >     defs.h          | 130 +++++++++++++++++++++++++++++++++++++++++++++++-
> > > > >     gdb-10.2.patch  | 110 +++++++++++++++++++++++++++++++++++++++-
> > > > >     gdb_interface.c |   2 +-
> > > > >     kernel.c        |  47 +++++++++++++++--
> > > > >     ppc64.c         |  95 +++++++++++++++++++++++++++++++++--
> > > > >     task.c          |  14 ++++++
> > > > >     tools.c         |   2 +-
> > > > >     8 files changed, 434 insertions(+), 10 deletions(-)
--
Crash-utility mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxxxxxx
https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
Contribution Guidelines: https://github.com/crash-utility/crash/wiki