Han, This is much worse than I thought. When you said "sometimes", you must mean "all of the time" with respect to the active tasks? Because that's what I see here. I provisioned a ppc64le machine, set up kdump to create a compressed kdump, and crashed the machine with sysrq-c. This is what I see with "bt -a": crash> bt -a PID: 12674 TASK: c00000002cc08810 CPU: 0 COMMAND: "bash" PID: 0 TASK: c0000001ee020000 CPU: 1 COMMAND: "swapper/1" PID: 0 TASK: c0000001ee021370 CPU: 2 COMMAND: "swapper/2" PID: 0 TASK: c0000001ee0226e0 CPU: 3 COMMAND: "swapper/3" PID: 0 TASK: c0000001ee023a50 CPU: 4 COMMAND: "swapper/4" PID: 0 TASK: c0000001ee024dc0 CPU: 5 COMMAND: "swapper/5" PID: 0 TASK: c0000001ee026130 CPU: 6 COMMAND: "swapper/6" PID: 0 TASK: c0000001ee0274a0 CPU: 7 COMMAND: "swapper/7" crash> As far as I can tell, it does not even use the registers in the notes, but rather defaults to searching the relevant stacks for ".crash_kexec". But on ppc64le, that symbol no longer exists as it does on a big-endian ppc64: crash> sym -q crash_kexec c000000000051760 (t) crash_kexec_prepare_cpus c0000000000519c0 (T) crash_kexec_secondary c00000000016e8a0 (T) crash_kexec crash> On the big-endian ppc64 machines, there was this construct where the text symbols started with a ".", and there were data symbols that pointed to them: crash> sym -q crash_kexec c000000000050050 (t) .crash_kexec_prepare_cpus c0000000000502b0 (T) .crash_kexec_secondary c000000000172160 (T) .crash_kexec c0000000012ab5f0 (d) crash_kexec_prepare_cpus c0000000012ab600 (D) crash_kexec_secondary c0000000012b9238 (D) crash_kexec crash> rd c0000000012b9238 c0000000012b9238: c000000000172160 ......!` crash> So therefore backtraces are guaranteed to fail for the active tasks on ppc64le, because the stack searching code does this: for (i = 0, up = (ulong *)bt->stackbuf; i < (bt->stacktop - bt->stackbase)/sizeof(ulong); i++, up++) { sym = closest_symbol(*up); if (STREQ(sym, ".netconsole_netdump") || STREQ(sym, ".netpoll_start_netdump") || STREQ(sym, ".start_disk_dump") || STREQ(sym, ".crash_kexec") || STREQ(sym, ".crash_fadump") || STREQ(sym, ".disk_dump")) { *nip = *up; *ksp = bt->stackbase + ((char *)(up) - 16 - bt->stackbuf); return TRUE; } } where no ".crash_kexec" symbol exists in ppc64le kernels. So Han, can you find out who in IBM should be responsible for supporting ppc64le in the crash utility? Or is it you? Thanks, Dave ----- Original Message ----- > > > ----- Original Message ----- > > > > ----- Original Message ----- > > > Hello, > > > > > > I just noticed that on ppc64le, sometimes "bt" cannot find the stack > > > info of current process. For example, there is a vmcore captured by > > > kdump on a ppc64le system, which running with a kernel version 3.10. The > > > vmcore was captured when kernel oopsed. There is no stack info found by > > > bt: > > > > Hello Han, > > > > I've never worked on the backtrace code for ppc64, as it was written > > by (and maintained by) IBM. From the debug messages, what happened is > > that the starting IP/SP hooks are not being found. The crash command > > sequence presumably looks like this: > > > > cmd_bt > > back_trace > > get_kdump_regs > > get_netdump_regs > > get_netdump_regs_ppc64 (should setup bt->machdep to point to > > NT_PRSTATUS note) > > ppc64_get_stack_frame > > ppc64_get_dumpfile_stack_frame > > ppc64_kdump_stack_frame (should get IP/SP pair based upon > > NT_PRSTATUS note contents) > > ppc64_back_trace_cmd > > ppc64_back_trace > > > > ppc64_kdump_stack_frame() should pull the starting NIP/KSP values from the > > pt_regs structure in the per-cpu NT_PRSTATUS note, but it appears that it > > is not, > > leaving the registers at their initialized values of NULL. > > > > This causes the failure later on when ppc64_back_trace_cmd() is called, and > > which > > prints the "=> PC: 0 () FP: 0" debug message shown below, and later on > > ppc64_back_trace() > > prints the "cannot find the stack info." debug message. > > > > Without the dumpfile, I can't offer much else. Can you verify the crash > > utility > > stack trail above, and if it is as I suspect, figure out why > > ppc64_kdump_stack_frame() > > is failing? Or what other path it is taking? > > Actually, if this is a compressed kdump, ppc64_kdump_stack_frame() will not > be > called, and the register access is done inside > ppc64_get_dumpfile_stack_frame(). > > The ppc64_get_dumpfile_stack_frame() function first grabs the registers from > the pt_regs > structure in the per-cpu NT_PRSTATUS note, but then also checks the hard and > soft IRQ > stacks, and the hardware interrupt stack, for known instances of kernel dump > functions, > which would override the pt_regs contents. If nothing is found on those > stacks, > the registers from the NT_PRSTATUS note are used. > > Dave > > > > > > > > > > > > crash 7.0.9-2.ael7b > > > Copyright (C) 2002-2014 Red Hat, Inc. > > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > > > Copyright (C) 1999-2006 Hewlett-Packard Co > > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > > Copyright (C) 2005, 2011 NEC Corporation > > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > This program is free software, covered by the GNU General Public License, > > > and you are welcome to change it and/or distribute copies of it under > > > certain conditions. Enter "help copying" to see the conditions. > > > This program has absolutely no warranty. Enter "help warranty" for > > > details. > > > > > > GNU gdb (GDB) 7.6 > > > Copyright (C) 2013 Free Software Foundation, Inc. > > > License GPLv3+: GNU GPL version 3 or later > > > <http://gnu.org/licenses/gpl.html> > > > This is free software: you are free to change and redistribute it. > > > There is NO WARRANTY, to the extent permitted by law. Type "show > > > copying" > > > and "show warranty" for details. > > > This GDB was configured as "powerpc64le-unknown-linux-gnu"... > > > > > > KERNEL: /usr/lib/debug/lib/modules/3.10.0-221.ael7b.ppc64le/vmlinux > > > DUMPFILE: /var/crash/127.0.0.1-2015.01.15-22:19:14/vmcore [PARTIAL > > > DUMP] > > > CPUS: 16 > > > DATE: Thu Jan 15 21:18:16 2015 > > > UPTIME: 17:53:43 > > > LOAD AVERAGE: 213.58, 213.23, 212.70 > > > TASKS: 1383 > > > NODENAME: thymelp2.isst.aus.stglabs.ibm.com > > > RELEASE: 3.10.0-221.ael7b.ppc64le > > > VERSION: #1 SMP Wed Jan 7 09:27:09 EST 2015 > > > MACHINE: ppc64le (3425 Mhz) > > > MEMORY: 15 GB > > > PANIC: "Oops: Kernel access of bad area, sig: 11 [#1]" (check log > > > for > > > details) > > > PID: 1970 > > > COMMAND: "cat" > > > TASK: c0000003130874a0 [THREAD_INFO: c00000005069c000] > > > CPU: 5 > > > STATE: TASK_RUNNING (PANIC) > > > > > > crash> set debug 99 > > > debug: 99 > > > crash> bt > > > PID: 1970 TASK: c0000003130874a0 CPU: 5 COMMAND: "cat" > > > GETBUF(16384 -> 0) > > > <readmem: c00000005069c000, KVADDR, "stack contents", 16384, (ROE), > > > 10a81570> > > > <read_diskdump: addr: c00000005069c000 paddr: 5069c000 cnt: 16384> > > > read_diskdump: paddr/pfn: 5069c000/5069 -> cache physical page: 50690000 > > > c00000005069c018: do_no_restart_syscall > > > c00000005069e870: blk_throtl_bio+240 > > > c00000005069e990: clone_endio > > > c00000005069ea00: generic_make_request_checks+836 > > > c00000005069eab8: hardware_interrupt_common+128 > > > c00000005069eac0: generic_make_request+36 > > > c00000005069eb10: mempool_alloc_slab+36 > > > c00000005069eb30: mempool_alloc+256 > > > c00000005069eb50: mempool_alloc_slab+36 > > > c00000005069ebc0: get_request+948 > > > c00000005069ec00: __split_and_process_bio+1408 > > > c00000005069ec20: autoremove_wake_function > > > c00000005069ec80: find_busiest_group+544 > > > c00000005069edf0: load_balance+684 > > > c00000005069ee10: blk_throtl_bio+240 > > > c00000005069ee70: find_busiest_group+544 > > > c00000005069eee0: dequeue_task_fair+968 > > > c00000005069ef30: clone_endio > > > c00000005069ef50: get_page_from_freelist+1436 > > > c00000005069f0a0: pSeries_cause_ipi_mux+112 > > > c00000005069f0c0: smp_send_reschedule+164 > > > c00000005069f0e0: default_wake_function+708 > > > c00000005069f160: __wake_up_locked+116 > > > c00000005069f1b0: ep_poll_callback+444 > > > c00000005069f250: run_posix_cpu_timers+104 > > > c00000005069f2c0: hvterm_raw_put_chars+64 > > > c00000005069f2e0: hvc_console_print+336 > > > c00000005069f3a8: initial_stab+2048 > > > c00000005069f3b0: crash_save_cpu+252 > > > c00000005069f488: cik_cp_resume+13476 > > > c00000005069f490: dev_get_drvdata > > > c00000005069f580: default_machine_kexec+332 > > > c00000005069f610: pSeries_machine_kexec+60 > > > c00000005069f680: machine_kexec+56 > > > c00000005069f6a0: crash_kexec+312 > > > c00000005069f6f0: dev_attr_show+64 > > > c00000005069f748: cik_cp_resume+13476 > > > c00000005069f750: dev_get_drvdata > > > c00000005069f7f0: radeon_hwmon_show_temp+72 > > > c00000005069f800: slb_miss_realmode+80 > > > c00000005069f808: dev_get_drvdata > > > c00000005069f810: radeon_hwmon_show_temp+32 > > > c00000005069f890: die+840 > > > c00000005069f930: bad_page_fault+224 > > > c00000005069f948: radeon_hwmon_show_temp+72 > > > c00000005069f9a0: handle_page_fault+44 > > > c00000005069fa00: dev_attr_show+64 > > > c00000005069fa58: cik_cp_resume+13476 > > > c00000005069fa60: dev_get_drvdata > > > c00000005069fb00: radeon_hwmon_show_temp+72 > > > c00000005069fb10: slb_miss_realmode+80 > > > c00000005069fb18: dev_get_drvdata > > > c00000005069fb20: radeon_hwmon_show_temp+32 > > > c00000005069fb60: handle_mm_fault+1724 > > > c00000005069fb80: sysfs_open_file > > > c00000005069fbd0: handle_page_fault+16 > > > c00000005069fc90: alloc_pages_current+416 > > > c00000005069fd00: dev_attr_show+64 > > > c00000005069fd30: sysfs_read_file+220 > > > c00000005069fde0: sys_read+304 > > > c00000005069fe40: syscall_exit > > > [3fffd0d6fe88] back_trace: > > > task: c0000003130874a0 > > > flags: 0 > > > instptr: 0 > > > stkptr: 0 > > > bptr: 0 > > > stackbase: c00000005069c000 > > > stacktop: c0000000506a0000 > > > tc: 1003c7b9fa8 (1970, c0000003130874a0) > > > hp: 0 > > > ref: 0 > > > stackbuf: 10a81570 > > > textlist: 0 > > > frameptr: 0 > > > call_target: none > > > eframe_ip: 0 > > > debug: 0 > > > radix: 0 > > > cpumask: 0 > > > => PC: 0 () FP: 0 > > > GETBUF(248 -> 1) > > > GETBUF(1500 -> 2) > > > cannot find the stack info. > > > FREEBUF(2) > > > FREEBUF(1) > > > crash> > > > > > > > > > Is this a problem? > > > > > > Thanks in advance! > > > > > > -- > > > Crash-utility mailing list > > > Crash-utility@xxxxxxxxxx > > > https://www.redhat.com/mailman/listinfo/crash-utility > > > > > > -- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility