Hi Alexey, On Tue, Mar 19, 2024 at 6:00 AM Alexey Makhalov <alexey.makhalov@xxxxxxxxxxxx> wrote: > > Hi Tao, > > I like the idea of having just one thread at a time from the gdb side, and it can be replaced by a "set" crash command. > Single semantics will work for both live and offline debugging and for both types of tasks: active or scheduled out. > Need to be careful with lifetime for the data (gdb caches). I'm experimenting with live debugging these days. > Aditya, Tao, let me know what is the latest patchset I should use from you folks? You can use https://github.com/liutgnu/crash-dev/commits/one-thread directly. The one-thread branch takes Aditya's v10 patchset and my v1 patchset, as well as the trial "one gdb thread/CPUS" patch. You can revert the trial one patch for testing. Thanks, Tao Liu > > Thanks, > --Alexey > > On Mon, Mar 18, 2024 at 2:56 AM Tao Liu <ltao@xxxxxxxxxx> wrote: >> >> Hi Aditya and Alexey, >> >> On Mon, Mar 18, 2024 at 11:35:41AM +0800, Tao Liu wrote: >> > On Sun, Mar 17, 2024 at 03:40:52PM +0530, Aditya Gupta wrote: >> > > Hi Tao and Alexey, >> > > >> > > On Fri, Mar 15, 2024 at 03:15:19PM -0700, Alexey Makhalov wrote: >> > > > Hi Tao, >> > > > >> > > > Thanks for the details. Agree, my approach is targeted primarily for >> > > > offline debugging. >> > > > Create/exit monitoring sounds like an overkill. And from the other side, >> > > > maintaining a static list of threads (CPUs or Tasks) does not work for live >> > > > debugging. >> > > > "ps" followed by "set" where GDB will receive just one inferior/thread at a >> > > > time is probably the right universal approach. >> > > > Let me experiment with a combination of my and your patches and come back >> > > > to you. >> > > >> > > Sorry I noticed a bug late while trying the combination of tao's and my >> > > patches. >> > > >> > > I agree with Tao that current approach might not be good for live >> > > debugging, but I think the approach of having all tasks in 'info >> > > threads' is a good advantage for offline debugging, since then gdb knows >> > > about all tasks (similar behaviour as usual gdb), and makes it easier >> > > to synchronise crash <-> gdb context. >> > > >> > > I had refrained from changing the format of 'info threads' from active >> > > tasks to all tasks, but there are some issues with crash <-> gdb sync, >> > > when doing arbitrary task unwinding. >> > > >> > > 'set 1' (let's assume it's an INACTIVE task on CPU 5) >> > > >> > > 'info threads' >> > > -> when it switches to CPU 5, it asks crash to sync, ie. `set_cpu(5)` >> > > -> crash will return CPU 5's active tasks registers >> > > -> WRONG registers for CPU 5, since we wanted registers for PID 1 (which >> > > is an inactive task on CPU 5) >> > >> > What do you mean of "WRONG registers"? Isn't the CPU5's regcache be pid >> > 1's context after 'set 1'? If it isn't then it's a bug. >> > >> > After crash finished loading and we see crash> prompt, the cpu regcaches >> > will be all active tasks', which are running on each cpu. That is what we saw >> > using "info threads", like: >> > >> > crash> info threads >> > Id Target Id Frame >> > .... >> > * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 >> > .... >> > 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 >> > .... >> > >> > Then we use "set <pid>" to switch to task <pid>, its cpu regcache will be >> > flushed, and if we use command "info threads" again, it will show the status >> > after the flushing. E.g in the following, task pid 1's cpu is #13, so >> > cpu #13 is flushed after "set 1": >> > >> > crash> ps >> > PID PPID CPU TASK ST %MEM VSZ RSS COMM >> > 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd >> > >> > crash> set 1 >> > crash> info threads >> > Id Target Id Frame >> > .... >> > 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 >> > .... >> > * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 >> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... >> > .... >> > >> > When design, I assume the CPUs/threads are like power slots, its number is >> > constant, but the contents can be changed. We don't need to prepare the >> > "slots" as many as our tasks quantity (I don't know the maxinum threads that >> > a gdb can accept, e.g. on a system which have 65K tasks, but for most >> > tasks we'd never switch to it in order to view its stack trace, I guess >> > it would be a waste to call add_thread_silent (target, ...) 65K times). >> > >> > Currently the gdb CPUs/threads(slots) equals to the system's cpu number. But >> > if a system which have thousands of cpus, which we think is unnecessary to >> > call add_thread_silent 1K times, we can even shrink the slots to be less. >> > >> >> I made a trial patch of leaving only 1 gdb CPUs/threads(slots) in [1]. >> And it can also view arbitrary tasks stack unwinding. In this case, >> "info threads" and "thread XX" will have no meaning to be kept. >> >> Please see my test results: >> >> KERNEL: /home/ltao/kernel/linux-next-next-20220914/vmlinux >> DUMPFILE: /proc/kcore >> CPUS: 4 >> ^^^^^^^ there are 4 cpus in the system >> crash> ps >> PID PPID CPU TASK ST %MEM VSZ RSS COMM >> ... >> 1 0 1 ffff9908002a6500 IN 0.1 253024 19444 systemd >> ... >> 5 2 0 ffff9908002a0000 ID 0.0 0 0 [netns] >> ... >> ^^^^^^^^^^^^^ we use pid 1 and pid 5, which belongs to 2 different cpus >> crash> set 1 >> crash> gdb bt >> #0 context_switch (rf=0xffffbf750001bd68, next=0xffff99080032cbc0, prev=0x0, rq=0xffff990b6fcafe80) at kernel/sched/core.c:5201 >> #1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510 >> #2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586 >> #3 0xffffffffb3cf6ab9 in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock_id=clock_id@entry=1) at kernel/time/hrtimer.c:2296 >> ... >> crash> bt >> PID: 1 TASK: ffff9908002a6500 CPU: 1 COMMAND: "systemd" >> #0 [ffffbf750001bdb0] schedule at ffffffffb3cf0603 >> #1 [ffffbf750001bdc0] schedule_hrtimeout_range_clock at ffffffffb3cf6ab9 >> ... >> ^^^ we can view pid 1's stack trace with no problem >> crash> info threads >> Id Target Id Frame >> * 1 CPU 0 context_switch (rf=0xffffbf750001bd68, next=0xffffffffb4c18a00 <init_task>, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201 >> >> crash> set 5 >> crash> gdb bt >> #0 context_switch (rf=0xffffbf750003be58, next=0xffff9908002d6500, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201 >> #1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510 >> #2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586 >> #3 0xffffffffb3308752 in rescuer_thread (__rescuer=0xffff9908001f7480) at kernel/workqueue.c:2599 >> ... >> crash> bt >> PID: 5 TASK: ffff9908002a0000 CPU: 0 COMMAND: "netns" >> #0 [ffffbf750003bea0] schedule at ffffffffb3cf0603 >> #1 [ffffbf750003beb0] rescuer_thread at ffffffffb3308752 >> ... >> ^^^ we can view pid 5's stack trace with no problem >> crash> info threads >> Id Target Id Frame >> * 1 CPU 0 context_switch (rf=0xffffbf750003be58, next=0xffff9908002d6500, prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201 >> ^^^^^^^^^^^^^^^^^^^^^ now its pid 5's context >> >> From my view, I think we can keep the current "gdb thread/CPUs = nr_cpus", or >> be more ambitious, leaving "gdb thread/CPUs = 1", in case there are >> systems with thousands of cpus, and we only need this in task.c: >> >> + gdb_refresh_regcache(0); >> - for (int i = 0; i < get_cpus_online(); i++) >> - gdb_refresh_regcache(i); >> >> Do you have any thoughts? >> >> [1]: https://github.com/liutgnu/crash-dev/commits/one-thread >> >> Thanks, >> Tao Liu >> >> > Thanks, >> > Tao Liu >> > >> > >> > > >> > > I can think of solving it in two ways: >> > > 1. Having a array in crash_target, which tells what CPU in gdb maps to >> > > which task in crash, this can be done without requiring changes to 'info >> > > threads', while we do the 'add_thread_silent' >> > > 2. Having all tasks in gdb itself, so it's a one-to-one, so `set_cpu` >> > > will then refer to the task ID, I will have to explore more on this. >> > > >> > > Any comments ? >> > > >> > > Thanks, >> > > Aditya Gupta >> > > >> > > > >> > > > Thanks, >> > > > --Alexey >> > > > >> > > > >> > > > On Thu, Mar 14, 2024 at 9:22 PM Tao Liu <ltao@xxxxxxxxxx> wrote: >> > > > >> > > > > Hi Alexey, >> > > > > >> > > > > On Thu, Mar 14, 2024 at 6:29 PM Alexey Makhalov >> > > > > <alexey.makhalov@xxxxxxxxxxxx> wrote: >> > > > > > >> > > > > > Support for GDB debugging of all tasks active and inactive. >> > > > > > Before this commit only active tasks were listed by "info threads" >> > > > > > with "CPU #" as a Target Id. >> > > > > > >> > > > > > "info threads" will now show all tasks, similar to "ps", example: >> > > > > > crash> info threads >> > > > > > Id Target Id Frame >> > > > > > * 1 0 swapper/0 0xffffffffadba19d4 in default_idle () at >> > > > > arch/x86/kernel/process.c:731 >> > > > > > 2 0 swapper/1 0xffffffffadba19d4 in default_idle () at >> > > > > arch/x86/kernel/process.c:731 >> > > > > > 3 0 swapper/2 0xffffffffadba19d4 in default_idle () at >> > > > > arch/x86/kernel/process.c:731 >> > > > > > 4 0 swapper/3 0xffffffffadba19d4 in default_idle () at >> > > > > arch/x86/kernel/process.c:731 >> > > > > > 5 0 swapper/4 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf0000f3e88, next=0xffff9ecb04908000, prev=<optimized out>, >> > > > > rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > ... >> > > > > > 730 970325 taskset 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf006a0fd18, next=0xffff9ecb0aec0000, prev=<optimized out>, >> > > > > rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > 731 975217 sleep 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized out>, >> > > > > rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > 732 975228 sleep 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf00696fb58, next=0xffff9ecac0690000, prev=<optimized out>, >> > > > > rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > ... >> > > > > > 876 976084 docker 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf0153dbd10, next=0xffff9ecac0645100, prev=<optimized out>, >> > > > > rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > 877 976085 systemd-userwor 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf0153cbc58, next=0xffff9ecac0645100, prev=<optimized out>, >> > > > > rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > 878 976086 systemd-userwor 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf0153e3c58, next=0xffffffffaec15a40 <init_task>, >> > > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > 879 976087 systemd-userwor 0xffffffffadb97292 in context_switch >> > > > > (rf=0xffffbaf0153ebc58, next=0xffffffffaec15a40 <init_task>, >> > > > > prev=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:5372 >> > > > > > Where "Target ID" contains "PID COMM" of the task >> > > > > > >> > > > > >> > > > > I see here, we chose a different tech path of viewing arbitrary tasks. >> > > > > You improved the "info threads" as an alternative to crash "ps" cmd, >> > > > > and use "thread X" to switch to task X and view its stacktrace. >> > > > > However mine is using crash "ps" cmd first, getting specific tasks pid >> > > > > or task struct, then using crash cmd "set <pid>" or "set >> > > > > <task_struct>" to switch to task X and view its stacktrace. >> > > > > >> > > > > I notice there is one problem of using the "info threads as ps" tech >> > > > > path, it is difficult to handle the live debug, see the comment below: >> > > > > >> > > > > > Example of "731 975217 sleep" debugging, real case, trying to >> > > > > > figure out why sleep was stuck in uninterruptable sleep. >> > > > > > Backtrace using crash: >> > > > > > crash> ps | grep 975217 >> > > > > > 975217 969797 3 ffff9ecb3956a880 UN 0.0 0 0 >> > > > > sleep >> > > > > > crash> bt 975217 >> > > > > > PID: 975217 TASK: ffff9ecb3956a880 CPU: 3 COMMAND: "sleep" >> > > > > > #0 [ffffbaf005743ba0] __schedule at ffffffffadb97292 >> > > > > > #1 [ffffbaf005743c60] schedule at ffffffffadb982b8 >> > > > > > #2 [ffffbaf005743c80] rwbase_write_lock at ffffffffadb9aed7 >> > > > > > #3 [ffffbaf005743cc0] down_write at ffffffffadb9b133 >> > > > > > #4 [ffffbaf005743cd0] unlink_file_vma at ffffffffad2b0e2e >> > > > > > #5 [ffffbaf005743cf8] free_pgtables at ffffffffad2a47b0 >> > > > > > #6 [ffffbaf005743d88] exit_mmap at ffffffffad2b3b8d >> > > > > > #7 [ffffbaf005743e80] mmput at ffffffffad08c81f >> > > > > > #8 [ffffbaf005743e98] do_exit at ffffffffad09636c >> > > > > > #9 [ffffbaf005743ef8] do_group_exit at ffffffffad096c78 >> > > > > > RIP: 00007f111c70ddf9 RSP: 00007fff451817e8 RFLAGS: 00000246 >> > > > > > RAX: ffffffffffffffda RBX: 00007f111c8089e0 RCX: 00007f111c70ddf9 >> > > > > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 >> > > > > > RBP: 0000000000000000 R8: ffffffffffffff80 R9: 0000000000000000 >> > > > > > R10: 00007fff451817b0 R11: 0000000000000246 R12: 00007f111c8089e0 >> > > > > > R13: 00007f111c80e2e0 R14: 0000000000000002 R15: 00007f111c80e2c8 >> > > > > > ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b >> > > > > > >> > > > > > Backtrace using gdb (pay attention, task must be selected by thread Id): >> > > > > > crash> thread 731 >> > > > > > [Switching to thread 731 ( 975217 sleep)] >> > > > > > 5372 switch_to(prev, next, prev); >> > > > > > crash> gdb bt >> > > > > > #0 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20, >> > > > > next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) at >> > > > > kernel/sched/core.c:5372 >> > > > > > #1 __schedule (sched_mode=sched_mode@entry=0) at >> > > > > kernel/sched/core.c:6696 >> > > > > > #2 0xffffffffadb982b8 in schedule () at kernel/sched/core.c:6772 >> > > > > > #3 0xffffffffadb9aed7 in rwbase_write_lock (rwb=rwb@entry=0xffff9ecaf1831430, >> > > > > state=state@entry=2) at kernel/locking/rwbase_rt.c:259 >> > > > > > #4 0xffffffffadb9b133 in __down_write (sem=sem@entry=0xffff9ecaf1831430) >> > > > > at kernel/locking/rwsem.c:1474 >> > > > > > #5 down_write (sem=sem@entry=0xffff9ecaf1831430) at >> > > > > kernel/locking/rwsem.c:1574 >> > > > > > #6 0xffffffffad2b0e2e in i_mmap_lock_write (mapping=<optimized out>) >> > > > > at ./include/linux/fs.h:466 >> > > > > > #7 unlink_file_vma (vma=vma@entry=0xffff9ecadd566090) at mm/mmap.c:127 >> > > > > > #8 0xffffffffad2a47b0 in free_pgtables (tlb=tlb@entry=0xffffbaf005743dd0, >> > > > > mt=mt@entry=0xffff9ecb28e3b180, vma=0xffff9ecadd566090, vma@entry=0xffff9ecadd566000, >> > > > > floor=floor@entry=0, ceiling=ceiling@entry=0) at mm/memory.c:431 >> > > > > > #9 0xffffffffad2b3b8d in exit_mmap (mm=mm@entry=0xffff9ecb28e3b180) >> > > > > at mm/mmap.c:3237 >> > > > > > #10 0xffffffffad08c81f in __mmput (mm=0xffff9ecb28e3b180) at >> > > > > kernel/fork.c:1204 >> > > > > > #11 mmput (mm=mm@entry=0xffff9ecb28e3b180) at kernel/fork.c:1226 >> > > > > > #12 0xffffffffad09636c in exit_mm () at kernel/exit.c:563 >> > > > > > #13 do_exit (code=code@entry=0) at kernel/exit.c:856 >> > > > > > #14 0xffffffffad096c78 in do_group_exit (exit_code=0) at >> > > > > kernel/exit.c:1019 >> > > > > > #15 0xffffffffad096cf8 in __do_sys_exit_group (error_code=<optimized >> > > > > out>) at kernel/exit.c:1030 >> > > > > > #16 __se_sys_exit_group (error_code=<optimized out>) at >> > > > > kernel/exit.c:1028 >> > > > > > #17 __x64_sys_exit_group (regs=<optimized out>) at kernel/exit.c:1028 >> > > > > > #18 0xffffffffadb8a327 in do_syscall_x64 (nr=<optimized out>, >> > > > > regs=0xffffbaf005743f58) at arch/x86/entry/common.c:51 >> > > > > > #19 do_syscall_64 (regs=0xffffbaf005743f58, nr=<optimized out>) at >> > > > > arch/x86/entry/common.c:81 >> > > > > > #20 0xffffffffadc000dc in entry_SYSCALL_64 () at >> > > > > arch/x86/entry/entry_64.S:120 >> > > > > > #21 0x00007f111c80e2c8 in ?? () >> > > > > > #22 0x0000000000000002 in ?? () >> > > > > > #23 0x00007f111c80e2e0 in ?? () >> > > > > > #24 0x00007f111c8089e0 in ?? () >> > > > > > #25 0x0000000000000000 in ?? () >> > > > > > crash> f 3 >> > > > > > 259 rwbase_schedule(); >> > > > > > crash> p *rwb >> > > > > > $1 = { >> > > > > > readers = { >> > > > > > counter = 1 >> > > > > > }, >> > > > > > rtmutex = { >> > > > > > wait_lock = { >> > > > > > raw_lock = { >> > > > > > { >> > > > > > val = { >> > > > > > counter = 0 >> > > > > > }, >> > > > > > { >> > > > > > locked = 0 '\000', >> > > > > > pending = 0 '\000' >> > > > > > }, >> > > > > > { >> > > > > > locked_pending = 0, >> > > > > > tail = 0 >> > > > > > } >> > > > > > } >> > > > > > } >> > > > > > }, >> > > > > > waiters = { >> > > > > > rb_root = { >> > > > > > rb_node = 0xffffbaf006977be0 >> > > > > > }, >> > > > > > rb_leftmost = 0xffffbaf00696fbe0 >> > > > > > }, >> > > > > > owner = 0xffff9ecb3956a881 >> > > > > > } >> > > > > > } >> > > > > > >> > > > > > Additional changes: >> > > > > > 1. Allow gdb "frame" command. >> > > > > > 2. Blacklist useless gdb "gcore" command. Use gcore plugin instead. >> > > > > > 3. Move crash_target_init() to later time as crash target requires a >> > > > > list of >> > > > > > tasks to be initialized. >> > > > > > >> > > > > > Known issues and TBD items: >> > > > > > 1. "info threads" may bail out first time throwing errors trying to >> > > > > access >> > > > > > userspace address during unwind process. Following "info threads" >> > > > > > invokations run without issues. >> > > > > > 2. To unwind a stack of inactive task, only modern Linux versions, which >> > > > > use >> > > > > > inactive_task_frame, are supported and only x86_64 architecture. >> > > > > > 3. gdb bt unwinder does not stop properly and may show invalid frames >> > > > > (21-25 >> > > > > > on example above). Not a regression, existed before. >> > > > > > 4. gdb bt unwinder does not work on active tasks in userspace. Not a >> > > > > regression, >> > > > > > existed before. >> > > > > > 5. Only x86_64 architecture supported. machdep->get_task_reg() must be >> > > > > > implemented for others. Not a regression, existed before. >> > > > > > 6. Active tasks registers fetching imlemented only for VMware dumps, see >> > > > > > x86_64_get_task_reg() for more details. Not a regression, existed >> > > > > before. >> > > > > > >> > > > > > Signed-off-by: Alexey Makhalov <alexey.makhalov@xxxxxxxxxxxx> >> > > > > > --- >> > > > > > crash_target.c | 39 ++++++++++++++++++++--------- >> > > > > > defs.h | 10 ++++++-- >> > > > > > gdb-10.2.patch | 7 ++---- >> > > > > > gdb_interface.c | 63 ++++++++++++++++++++++++++++++---------------- >> > > > > > help.c | 1 + >> > > > > > main.c | 1 + >> > > > > > task.c | 1 + >> > > > > > x86_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++------ >> > > > > > 8 files changed, 140 insertions(+), 48 deletions(-) >> > > > > > >> > > > > > diff --git a/crash_target.c b/crash_target.c >> > > > > > index 4554806..2fdf203 100644 >> > > > > > --- a/crash_target.c >> > > > > > +++ b/crash_target.c >> > > > > > @@ -2,6 +2,8 @@ >> > > > > > * crash_target.c >> > > > > > * >> > > > > > * Copyright (c) 2021 VMware, Inc. >> > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" >> > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. >> > > > > > * >> > > > > > * This program is free software; you can redistribute it and/or modify >> > > > > > * it under the terms of the GNU General Public License as published by >> > > > > > @@ -13,7 +15,7 @@ >> > > > > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> > > > > > * GNU General Public License for more details. >> > > > > > * >> > > > > > - * Author: Alexey Makhalov <amakhalov@xxxxxxxxxx> >> > > > > > + * Author: Alexey Makhalov <alexey.makhalov@xxxxxxxxxxxx> >> > > > > > */ >> > > > > > >> > > > > > #include <defs.h> >> > > > > > @@ -23,11 +25,11 @@ >> > > > > > #include "regcache.h" >> > > > > > #include "gdbarch.h" >> > > > > > >> > > > > > -void crash_target_init (void); >> > > > > > - >> > > > > > +extern "C" void crash_target_init (void); >> > > > > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, int); >> > > > > > -extern "C" int crash_get_nr_cpus(void); >> > > > > > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char >> > > > > *regname, >> > > > > > +extern "C" int crash_get_nr_tasks(void); >> > > > > > +extern "C" void crash_get_task_info(int task_nr, unsigned long *pid, >> > > > > char **comm); >> > > > > > +extern "C" int crash_get_task_reg (int task_nr, int regno, const char >> > > > > *regname, >> > > > > > int regsize, void *val); >> > > > > > >> > > > > > >> > > > > > @@ -60,7 +62,13 @@ public: >> > > > > > bool has_registers () override { return true; } >> > > > > > bool thread_alive (ptid_t ptid) override { return true; } >> > > > > > std::string pid_to_str (ptid_t ptid) override >> > > > > > - { return string_printf ("CPU %ld", ptid.tid ()); } >> > > > > > + { >> > > > > > + unsigned long pid; >> > > > > > + char *comm; >> > > > > > + >> > > > > > + crash_get_task_info(ptid.tid(), &pid, &comm); >> > > > > > + return string_printf ("%7ld %s", pid, comm); >> > > > > > + } >> > > > > > >> > > > > > }; >> > > > > > >> > > > > > @@ -68,18 +76,25 @@ public: >> > > > > > void >> > > > > > crash_target::fetch_registers (struct regcache *regcache, int regno) >> > > > > > { >> > > > > > + int r; >> > > > > > gdb_byte regval[16]; >> > > > > > - int cpu = inferior_ptid.tid(); >> > > > > > + int task_nr = inferior_ptid.tid(); >> > > > > > struct gdbarch *arch = regcache->arch (); >> > > > > > >> > > > > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) >> > > > > > + if (regno >= 0) { >> > > > > > + r = regno; >> > > > > > + goto onetime; >> > > > > > + } >> > > > > > + >> > > > > > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) >> > > > > > { >> > > > > > +onetime: >> > > > > > const char *regname = gdbarch_register_name(arch, r); >> > > > > > int regsize = register_size (arch, r); >> > > > > > if (regsize > sizeof (regval)) >> > > > > > error (_("fatal error: buffer size is not enough to fit >> > > > > register value")); >> > > > > > >> > > > > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void *)®val)) >> > > > > > + if (crash_get_task_reg (task_nr, r, regname, regsize, (void >> > > > > *)®val)) >> > > > > > regcache->raw_supply (r, regval); >> > > > > > else >> > > > > > regcache->raw_supply (r, NULL); >> > > > > > @@ -107,10 +122,10 @@ crash_target::xfer_partial (enum target_object >> > > > > object, const char *annex, >> > > > > > >> > > > > > #define CRASH_INFERIOR_PID 1 >> > > > > > >> > > > > > -void >> > > > > > +extern "C" void >> > > > > > crash_target_init (void) >> > > > > > { >> > > > > > - int nr_cpus = crash_get_nr_cpus(); >> > > > > > + int nr_tasks = crash_get_nr_tasks(); >> > > > > > crash_target *target = new crash_target (); >> > > > > > >> > > > > > /* Own the target until it is successfully pushed. */ >> > > > > > @@ -119,7 +134,7 @@ crash_target_init (void) >> > > > > > push_target (std::move (target_holder)); >> > > > > > >> > > > > > inferior_appeared (current_inferior (), CRASH_INFERIOR_PID); >> > > > > > - for (int i = 0; i < nr_cpus; i++) >> > > > > > + for (int i = 0; i < nr_tasks; i++) >> > > > > > { >> > > > > > thread_info *thread = add_thread_silent (target, >> > > > > > ptid_t(CRASH_INFERIOR_PID, 0, >> > > > > i)); >> > > > > > diff --git a/defs.h b/defs.h >> > > > > > index 98650e8..2b3f247 100644 >> > > > > > --- a/defs.h >> > > > > > +++ b/defs.h >> > > > > > @@ -1080,7 +1080,7 @@ struct machdep_table { >> > > > > > void (*get_irq_affinity)(int); >> > > > > > void (*show_interrupts)(int, ulong *); >> > > > > > int (*is_page_ptr)(ulong, physaddr_t *); >> > > > > > - int (*get_cpu_reg)(int, int, const char *, int, void *); >> > > > > > + int (*get_task_reg)(struct task_context *, int, const char *, >> > > > > int, void *); >> > > > > > int (*is_cpu_prstatus_valid)(int cpu); >> > > > > > }; >> > > > > > >> > > > > > @@ -2263,6 +2263,7 @@ struct size_table { /* stash of >> > > > > commonly-used sizes */ >> > > > > > long pt_regs; >> > > > > > long task_struct; >> > > > > > long thread_info; >> > > > > > + long inactive_task_frame; >> > > > > > long softirq_state; >> > > > > > long desc_struct; >> > > > > > long umode_t; >> > > > > > @@ -8001,9 +8002,14 @@ extern int have_full_symbols(void); >> > > > > > #define XEN_HYPERVISOR_ARCH >> > > > > > #endif >> > > > > > >> > > > > > +/* >> > > > > > + * crash_target.c >> > > > > > + */ >> > > > > > +extern void crash_target_init (void); >> > > > > > + >> > > > > > /* >> > > > > > * Register numbers must be in sync with gdb/features/i386/64bit-core.c >> > > > > > - * to make crash_target->fetch_registers() ---> machdep->get_cpu_reg() >> > > > > > + * to make crash_target->fetch_registers() ---> machdep->get_task_reg() >> > > > > > * working properly. >> > > > > > */ >> > > > > > enum x86_64_regnum { >> > > > > > diff --git a/gdb-10.2.patch b/gdb-10.2.patch >> > > > > > index a7018a2..ecf673d 100644 >> > > > > > --- a/gdb-10.2.patch >> > > > > > +++ b/gdb-10.2.patch >> > > > > > @@ -221,7 +221,7 @@ exit 0 >> > > > > > warning (_("\ >> > > > > > --- gdb-10.2/gdb/main.c.orig >> > > > > > +++ gdb-10.2/gdb/main.c >> > > > > > -@@ -392,6 +392,14 @@ start_event_loop () >> > > > > > +@@ -392,6 +392,13 @@ start_event_loop () >> > > > > > return; >> > > > > > } >> > > > > > >> > > > > > @@ -230,7 +230,6 @@ exit 0 >> > > > > > +extern "C" void main_loop(void); >> > > > > > +extern "C" unsigned long crash_get_kaslr_offset(void); >> > > > > > +extern "C" int console(const char *, ...); >> > > > > > -+void crash_target_init (void); >> > > > > > +#endif >> > > > > > + >> > > > > > /* Call command_loop. */ >> > > > > > @@ -316,7 +315,7 @@ exit 0 >> > > > > > } >> > > > > > } >> > > > > > >> > > > > > -@@ -1242,6 +1274,16 @@ captured_main (void *data) >> > > > > > +@@ -1242,6 +1274,14 @@ captured_main (void *data) >> > > > > > >> > > > > > captured_main_1 (context); >> > > > > > >> > > > > > @@ -324,8 +323,6 @@ exit 0 >> > > > > > + /* Relocate the vmlinux. */ >> > > > > > + objfile_rebase (symfile_objfile, crash_get_kaslr_offset()); >> > > > > > + >> > > > > > -+ crash_target_init(); >> > > > > > -+ >> > > > > > + /* Back to crash. */ >> > > > > > + main_loop(); >> > > > > > +#endif >> > > > > > diff --git a/gdb_interface.c b/gdb_interface.c >> > > > > > index b14319c..03178f5 100644 >> > > > > > --- a/gdb_interface.c >> > > > > > +++ b/gdb_interface.c >> > > > > > @@ -3,6 +3,8 @@ >> > > > > > * Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. >> > > > > > * Copyright (C) 2002-2015,2018-2019 David Anderson >> > > > > > * Copyright (C) 2002-2015,2018-2019 Red Hat, Inc. All rights reserved. >> > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term "Broadcom" >> > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. >> > > > > > * >> > > > > > * This program is free software; you can redistribute it and/or modify >> > > > > > * it under the terms of the GNU General Public License as published by >> > > > > > @@ -711,7 +713,7 @@ static char *prohibited_list[] = { >> > > > > > "watch", "rwatch", "awatch", "attach", "continue", "c", "fg", >> > > > > "detach", >> > > > > > "finish", "handle", "interrupt", "jump", "kill", "next", "nexti", >> > > > > > "signal", "step", "s", "stepi", "target", "until", "delete", >> > > > > > - "clear", "disable", "enable", "condition", "ignore", "frame", >> > > > > "catch", >> > > > > > + "clear", "disable", "enable", "condition", "ignore", "gcore", >> > > > > "catch", >> > > > > > "tcatch", "return", "file", "exec-file", "core-file", >> > > > > "symbol-file", >> > > > > > "load", "si", "ni", "shell", "sy", >> > > > > > NULL /* must be last */ >> > > > > > @@ -877,6 +879,7 @@ gdb_readmem_callback(ulong addr, void *buf, int len, >> > > > > int write) >> > > > > > switch (len) >> > > > > > { >> > > > > > case SIZEOF_8BIT: >> > > > > > + fprintf(fp, "%s\n", pc->curcmd); >> > > > > > if (STREQ(pc->curcmd, "bt")) { >> > > > > > if (readmem(addr, memtype, buf, SIZEOF_8BIT, >> > > > > > "gdb_readmem_callback", readflags)) >> > > > > > @@ -1063,34 +1066,52 @@ get_frame_offset(ulong pc) >> > > > > > unsigned long crash_get_kaslr_offset(void); >> > > > > > unsigned long crash_get_kaslr_offset(void) >> > > > > > { >> > > > > > - return kt->relocate * -1; >> > > > > > + return kt->relocate * -1; >> > > > > > } >> > > > > > >> > > > > > /* Callbacks for crash_target */ >> > > > > > -int crash_get_nr_cpus(void); >> > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, >> > > > > > +int crash_get_nr_tasks(void); >> > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm); >> > > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, >> > > > > > int regsize, void *val); >> > > > > > >> > > > > > -int crash_get_nr_cpus(void) >> > > > > > +int crash_get_nr_tasks(void) >> > > > > > { >> > > > > > - if (SADUMP_DUMPFILE()) >> > > > > > - return sadump_get_nr_cpus(); >> > > > > > - else if (DISKDUMP_DUMPFILE()) >> > > > > > - return diskdump_get_nr_cpus(); >> > > > > > - else if (KDUMP_DUMPFILE()) >> > > > > > - return kdump_get_nr_cpus(); >> > > > > > - else if (VMSS_DUMPFILE()) >> > > > > > - return vmware_vmss_get_nr_cpus(); >> > > > > > - >> > > > > > - /* Just CPU #0 */ >> > > > > ()> - return 1; >> > > > > > + return RUNNING_TASKS(); >> > > > > > } >> > > > > >> > > > > The crash_get_nr_tasks() will return a fixed number of running tasks, >> > > > > which is the task status when crash is loading. However for live debug >> > > > > mode, there will always be tasks exit and create, so we need to sync >> > > > > the tasks number with the "info threads". I guess it is not an easy >> > > > > work because that will involve: >> > > > > >> > > > > when crash loading: >> > > > > for (int i = 0; i < nr_tasks; i++) >> > > > > { >> > > > > thread_info *thread = add_thread_silent (target, ... >> > > > > } >> > > > > >> > > > > when new task create: >> > > > > thread_info *thread = add_thread_silent (target, ... >> > > > > >> > > > > when task exit: >> > > > > delete_thread(...) >> > > > > >> > > > > We may monitor those task create/exit events and do thread add/delete >> > > > > in a callback function or we just always do all tasks delete/add when >> > > > > "info thread" is invoked. But I guess none of these are as simple as >> > > > > using crash "ps" cmd and "set <pid>", there won't be any problem. >> > > > > Because crash "ps" can always show the current tasks status, and set >> > > > > <pid> can switch the task context to it. >> > > > > >> > > > > What do you think? >> > > > > >> > > > > Thanks, >> > > > > Tao Liu >> > > > > >> > > > > > >> > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, >> > > > > > - int regsize, void *value) >> > > > > > +/* Get task information by its index number in TT */ >> > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char **comm) >> > > > > > { >> > > > > > - if (!machdep->get_cpu_reg) >> > > > > > - return FALSE; >> > > > > > - return machdep->get_cpu_reg(cpu, regno, regname, regsize, >> > > > > value); >> > > > > > + int i; >> > > > > > + struct task_context *tc; >> > > > > > + >> > > > > > + tc = FIRST_CONTEXT(); >> > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) >> > > > > > + if (i == task_nr) { >> > > > > > + *pid = tc->pid; >> > > > > > + *comm = tc->comm; >> > > > > > + return; >> > > > > > + } >> > > > > > + *pid = 0; >> > > > > > + *comm = NULL; >> > > > > > + return; >> > > > > > +} >> > > > > > + >> > > > > > +int crash_get_task_reg (int task_nr, int regno, const char *regname, >> > > > > > + int regsize, void *value) >> > > > > > +{ >> > > > > > + int i; >> > > > > > + struct task_context *tc; >> > > > > > + >> > > > > > + if (!machdep->get_task_reg) >> > > > > > + return FALSE; >> > > > > > + >> > > > > > + tc = FIRST_CONTEXT(); >> > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) >> > > > > > + if (i == task_nr) { >> > > > > > + return machdep->get_task_reg(tc, regno, regname, >> > > > > regsize, value); >> > > > > > + } >> > > > > > + return FALSE; >> > > > > > } >> > > > > > >> > > > > > diff --git a/help.c b/help.c >> > > > > > index a9c4d30..85dbda5 100644 >> > > > > > --- a/help.c >> > > > > > +++ b/help.c >> > > > > > @@ -8520,6 +8520,7 @@ char *version_info[] = { >> > > > > > "Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.", >> > > > > > "Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.", >> > > > > > "Copyright (C) 2015, 2021 VMware, Inc.", >> > > > > > +"Copyright (C) 2024 Broadcom, Inc.", >> > > > > > "This program is free software, covered by the GNU General Public >> > > > > License,", >> > > > > > "and you are welcome to change it and/or distribute copies of it under", >> > > > > > "certain conditions. Enter \"help copying\" to see the conditions.", >> > > > > > diff --git a/main.c b/main.c >> > > > > > index 0b6b927..13acd2d 100644 >> > > > > > --- a/main.c >> > > > > > +++ b/main.c >> > > > > > @@ -794,6 +794,7 @@ main_loop(void) >> > > > > > } else >> > > > > > SIGACTION(SIGINT, restart, &pc->sigaction, NULL); >> > > > > > >> > > > > > + crash_target_init(); >> > > > > > /* >> > > > > > * Display system statistics and current context. >> > > > > > */ >> > > > > > diff --git a/task.c b/task.c >> > > > > > index ebdb5be..5d26c52 100644 >> > > > > > --- a/task.c >> > > > > > +++ b/task.c >> > > > > > @@ -298,6 +298,7 @@ task_init(void) >> > > > > > tt->flags |= THREAD_INFO; >> > > > > > } >> > > > > > >> > > > > > + STRUCT_SIZE_INIT(inactive_task_frame, "inactive_task_frame"); >> > > > > > MEMBER_OFFSET_INIT(task_struct_state, "task_struct", "state"); >> > > > > > MEMBER_SIZE_INIT(task_struct_state, "task_struct", "state"); >> > > > > > if (INVALID_MEMBER(task_struct_state)) { >> > > > > > diff --git a/x86_64.c b/x86_64.c >> > > > > > index 502817d..b6e36a5 100644 >> > > > > > --- a/x86_64.c >> > > > > > +++ b/x86_64.c >> > > > > > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct bt_info *, >> > > > > ulong, ulong, char *); >> > > > > > static void x86_64_framesize_debug(struct bt_info *); >> > > > > > static void x86_64_get_active_set(void); >> > > > > > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); >> > > > > > -static int x86_64_get_cpu_reg(int, int, const char *, int, void *); >> > > > > > +static int x86_64_get_task_reg(struct task_context *, int, const char >> > > > > *, int, void *); >> > > > > > static int x86_64_verify_paddr(uint64_t); >> > > > > > static void GART_init(void); >> > > > > > static void x86_64_exception_stacks_init(void); >> > > > > > @@ -195,7 +195,7 @@ x86_64_init(int when) >> > > > > > machdep->machspec->irq_eframe_link = UNINITIALIZED; >> > > > > > machdep->machspec->irq_stack_gap = UNINITIALIZED; >> > > > > > machdep->get_kvaddr_ranges = x86_64_get_kvaddr_ranges; >> > > > > > - machdep->get_cpu_reg = x86_64_get_cpu_reg; >> > > > > > + machdep->get_task_reg = x86_64_get_task_reg; >> > > > > > if (machdep->cmdline_args[0]) >> > > > > > parse_cmdline_args(); >> > > > > > if ((string = pc->read_vmcoreinfo("relocate"))) { >> > > > > > @@ -891,7 +891,7 @@ x86_64_dump_machdep_table(ulong arg) >> > > > > > fprintf(fp, " is_page_ptr: x86_64_is_page_ptr()\n"); >> > > > > > fprintf(fp, " verify_paddr: x86_64_verify_paddr()\n"); >> > > > > > fprintf(fp, " get_kvaddr_ranges: >> > > > > x86_64_get_kvaddr_ranges()\n"); >> > > > > > - fprintf(fp, " get_cpu_reg: x86_64_get_cpu_reg()\n"); >> > > > > > + fprintf(fp, " get_task_reg: x86_64_get_task_reg()\n"); >> > > > > > fprintf(fp, " init_kernel_pgd: x86_64_init_kernel_pgd()\n"); >> > > > > > fprintf(fp, "clear_machdep_cache: >> > > > > x86_64_clear_machdep_cache()\n"); >> > > > > > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? >> > > > > > @@ -6398,6 +6398,9 @@ x86_64_ORC_init(void) >> > > > > > }; >> > > > > > struct ORC_data *orc; >> > > > > > >> > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_bp, >> > > > > "inactive_task_frame", "bp"); >> > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, >> > > > > "inactive_task_frame", "ret_addr"); >> > > > > > + >> > > > > > if (machdep->flags & FRAMEPOINTER) >> > > > > > return; >> > > > > > >> > > > > > @@ -6455,9 +6458,6 @@ x86_64_ORC_init(void) >> > > > > > orc->__stop_orc_unwind = symbol_value("__stop_orc_unwind"); >> > > > > > orc->orc_lookup = symbol_value("orc_lookup"); >> > > > > > >> > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_bp, >> > > > > "inactive_task_frame", "bp"); >> > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, >> > > > > "inactive_task_frame", "ret_addr"); >> > > > > > - >> > > > > > orc->has_signal = MEMBER_EXISTS("orc_entry", "signal"); /* added >> > > > > at 6.3 */ >> > > > > > orc->has_end = MEMBER_EXISTS("orc_entry", "end"); /* >> > > > > removed at 6.4 */ >> > > > > > >> > > > > > @@ -9070,14 +9070,64 @@ x86_64_get_kvaddr_ranges(struct vaddr_range *vrp) >> > > > > > } >> > > > > > >> > > > > > static int >> > > > > > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, >> > > > > > +x86_64_get_task_reg(struct task_context *tc, int regno, const char >> > > > > *name, >> > > > > > int size, void *value) >> > > > > > { >> > > > > > if (regno >= LAST_REGNUM) >> > > > > > return FALSE; >> > > > > > >> > > > > > + /* >> > > > > > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 and r15 >> > > > > from >> > > > > > + * inactive_task_frame (see __switch_to_asm). Other regs saved on >> > > > > > + * regular frame. >> > > > > > + */ >> > > > > > + if (!is_task_active(tc->task)) { >> > > > > > + int frame_size = STRUCT_SIZE("inactive_task_frame"); >> > > > > > + >> > > > > > + /* Only modern kernels supported. */ >> > > > > > + if (tt->flags & THREAD_INFO && frame_size == 7 * 8) { >> > > > > > + ulong rsp; >> > > > > > + int offset = 0; >> > > > > > + switch (regno) { >> > > > > > + case RSP_REGNUM: >> > > > > > + readmem(tc->task + >> > > > > OFFSET(task_struct_thread) + >> > > > > > + >> > > > > OFFSET(thread_struct_rsp), KVADDR, >> > > > > > + &rsp, sizeof(void *), >> > > > > > + "thread_struct rsp", >> > > > > FAULT_ON_ERROR); >> > > > > > + rsp += frame_size; >> > > > > > + memcpy(value, &rsp, size); >> > > > > > + return TRUE; >> > > > > > + case RIP_REGNUM: >> > > > > > + offset += 8; >> > > > > > + case RBP_REGNUM: >> > > > > > + offset += 8; >> > > > > > + case RBX_REGNUM: >> > > > > > + offset += 8; >> > > > > > + case R12_REGNUM: >> > > > > > + offset += 8; >> > > > > > + case R13_REGNUM: >> > > > > > + offset += 8; >> > > > > > + case R14_REGNUM: >> > > > > > + offset += 8; >> > > > > > + case R15_REGNUM: >> > > > > > + readmem(tc->task + >> > > > > OFFSET(task_struct_thread) + >> > > > > > + >> > > > > OFFSET(thread_struct_rsp), KVADDR, >> > > > > > + &rsp, sizeof(void *), >> > > > > > + "thread_struct rsp", >> > > > > FAULT_ON_ERROR); >> > > > > > + readmem(rsp + offset, KVADDR, >> > > > > value, sizeof(void *), >> > > > > > + >> > > > > "inactive_thread_frame saved regs", FAULT_ON_ERROR); >> > > > > > + return TRUE; >> > > > > > + } >> > > > > > + } >> > > > > > + /* TBD: older kernels support. */ >> > > > > > + return FALSE; >> > > > > > + } >> > > > > > + >> > > > > > + /* >> > > > > > + * Task is active, grab CPU's registers >> > > > > > + */ >> > > > > > if (VMSS_DUMPFILE()) >> > > > > > - return vmware_vmss_get_cpu_reg(cpu, regno, name, size, >> > > > > value); >> > > > > > + return vmware_vmss_get_cpu_reg(tc->processor, regno, >> > > > > name, size, value); >> > > > > > >> > > > > > return FALSE; >> > > > > > } >> > > > > > -- >> > > > > > 2.39.0 >> > > > > > >> > > > > >> > > > > >> > > >> -- Crash-utility mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxxxxxx https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/ Contribution Guidelines: https://github.com/crash-utility/crash/wiki
