On Fri, Nov 20, 2015 at 04:25:38PM -0500, Dave Anderson wrote: > > > ----- Original Message ----- > > On Fri, Nov 20, 2015 at 03:18:55PM -0500, Dave Anderson wrote: > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > > ----- Original Message ----- > > > > > QEMU can generate both non-makedumpfile (just elf) and makedumpfile > > > > > formatted kdumps. In neither case will crash_notes have prstatus, as > > > > > crash_kexec doesn't run in the kernel, however the elf notes will > > > > > contain the prstatus, and we can dig them out of there. > > > > > > > > I don't have a lot of ARM and ARM64 dumpfiles, but just doing a > > > > quick sanity test of your patch, I came across this ARM dumpfile, > > > > which I believe may be a QEMU-generated ELF vmcore. I'm not sure, > > > > but it only has 1 NT_PRSTATUS note for the 1 online cpu (of 5 cpus). > > > > If it's more than a day old then it won't be from qemu. I just posted > > the patches for that yesterday morning :-) > > Even for 32-bit ARM? I honestly can't remember where/who it came from, Yup. The qemu patches I posted are for both arm and aarch64. Before those patches it wasn't possible to generate a dump with qemu for either. > although it's in my notes somewhere. Because the non-panicking cpus > are offline, it's possible that it's a kdump-generated dumpfile, and > the other cpu's note were not captured. I don't know, but regardless, > it turns out to be a damn good test case! > > > > > > > > > > > But anyway, note that as expected, it cannot find the registers in the > > > > kernel's uninitialized crash_notes -- here without your patch: > > > > So it looks like there may be other dump types (besides qemu generated) > > that can result in missing crash_notes. I don't know what those are, other > > than just corruption? In any case, I think this dump is still a good test > > case for the reason you found below. > > > > > > > > > > $ crash vmcore.pae vmlinux.pae.gz > > > > > > > > crash 7.1.4rc15 > > > > Copyright (C) 2002-2014 Red Hat, Inc. > > > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > > > > Copyright (C) 1999-2006 Hewlett-Packard Co > > > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > > > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > > > Copyright (C) 2005, 2011 NEC Corporation > > > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > > This program is free software, covered by the GNU General Public > > > > License, > > > > and you are welcome to change it and/or distribute copies of it under > > > > certain conditions. Enter "help copying" to see the conditions. > > > > This program has absolutely no warranty. Enter "help warranty" for > > > > details. > > > > > > > > GNU gdb (GDB) 7.6 > > > > Copyright (C) 2013 Free Software Foundation, Inc. > > > > License GPLv3+: GNU GPL version 3 or later > > > > <http://gnu.org/licenses/gpl.html> > > > > This is free software: you are free to change and redistribute it. > > > > There is NO WARRANTY, to the extent permitted by law. Type "show > > > > copying" > > > > and "show warranty" for details. > > > > This GDB was configured as "--host=x86_64-unknown-linux-gnu > > > > --target=arm-elf-linux"... > > > > > > > > WARNING: invalid note (n_type != NT_PRSTATUS) > > > > WARNING: cannot retrieve registers for active tasks > > > > > > > > KERNEL: vmlinux.pae.gz > > > > DUMPFILE: vmcore.pae > > > > CPUS: 5 [OFFLINE: 4] > > > > DATE: Sun Jun 8 18:27:39 2014 > > > > UPTIME: 00:03:22 > > > > LOAD AVERAGE: 0.16, 0.16, 0.07 > > > > TASKS: 51 > > > > NODENAME: buildroot > > > > RELEASE: 3.13.5 > > > > VERSION: #3 SMP Mon Jun 9 05:58:39 CST 2014 > > > > MACHINE: armv7l (unknown Mhz) > > > > MEMORY: 256 MB > > > > PANIC: "SysRq : Trigger a crash" > > > > PID: 732 > > > > COMMAND: "sh" > > > > TASK: 8bcead00 [THREAD_INFO: 8ad32000] > > > > CPU: 0 > > > > STATE: TASK_RUNNING (SYSRQ) > > > > > > > > crash> bt -a > > > > PID: 732 TASK: 8bcead00 CPU: 0 COMMAND: "sh" > > > > bt: WARNING: cannot determine starting stack frame for task 8bcead00 > > > > > > > > PID: 0 TASK: 8bc561c0 CPU: 1 COMMAND: "swapper/1" > > > > bt: WARNING: cannot determine starting stack frame for task 8bc561c0 > > > > > > > > PID: 0 TASK: 8bc56580 CPU: 2 COMMAND: "swapper/2" > > > > bt: WARNING: cannot determine starting stack frame for task 8bc56580 > > > > > > > > PID: 0 TASK: 8bc56940 CPU: 3 COMMAND: "swapper/3" > > > > bt: WARNING: cannot determine starting stack frame for task 8bc56940 > > > > > > > > PID: 0 TASK: 8bc56d00 CPU: 4 COMMAND: "swapper/4" > > > > bt: WARNING: cannot determine starting stack frame for task 8bc56d00 > > > > crash> > > > > > > > > > > > > With your patch applied, it generates a SIGSEGV in arm_get_crash_notes(): > > > > > > > > $ ./crash /usr/dumps/ARM/vmcore.pae /usr/dumps/ARM/vmlinux.pae.gz > > > > > > > > crash 7.1.4rc15 > > > > Copyright (C) 2002-2014 Red Hat, Inc. > > > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > > > > Copyright (C) 1999-2006 Hewlett-Packard Co > > > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > > > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > > > Copyright (C) 2005, 2011 NEC Corporation > > > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > > This program is free software, covered by the GNU General Public > > > > License, > > > > and you are welcome to change it and/or distribute copies of it under > > > > certain conditions. Enter "help copying" to see the conditions. > > > > This program has absolutely no warranty. Enter "help warranty" for > > > > details. > > > > > > > > GNU gdb (GDB) 7.6 > > > > Copyright (C) 2013 Free Software Foundation, Inc. > > > > License GPLv3+: GNU GPL version 3 or later > > > > <http://gnu.org/licenses/gpl.html> > > > > This is free software: you are free to change and redistribute it. > > > > There is NO WARRANTY, to the extent permitted by law. Type "show > > > > copying" > > > > and "show warranty" for details. > > > > This GDB was configured as "--host=x86_64-unknown-linux-gnu > > > > --target=arm-elf-linux"... > > > > > > > > Segmentation fault (core dumped) > > > > $ > > > > > > > > I haven't debugged it other than determining that the "note" it looks to > > > > have found the single note OK, but then upon continuation the next time > > > > through the loop, the "note" pointer is valid at line 597, but your > > > > function sets it back to NULL, and therefore it craps out at line 622: > > > > > > > > 597 note = (Elf32_Nhdr *)buf; > > > > 598 p = buf + sizeof(Elf32_Nhdr); > > > > 599 > > > > 600 /* > > > > 601 * dumpfiles created with qemu won't have > > > > crash_notes, but there will > > > > 602 * be elf notes. > > > > 603 */ > > > > 604 if (note->n_namesz == 0 && (DISKDUMP_DUMPFILE() > > > > || > > > > KDUMP_DUMPFILE())) { > > > > 605 if (DISKDUMP_DUMPFILE()) > > > > 606 note = > > > > diskdump_get_prstatus_percpu(i); > > > > 607 else if (KDUMP_DUMPFILE()) > > > > 608 note = > > > > netdump_get_prstatus_percpu(i); > > > > 609 if (note) { > > > > 610 /* > > > > 611 * SIZE(note_buf) accounts for a > > > > "final note", which is a > > > > 612 * trailing empty elf note > > > > header. > > > > 613 */ > > > > 614 long notesz = SIZE(note_buf) - > > > > sizeof(Elf32_Nhdr); > > > > 615 > > > > 616 if (sizeof(Elf32_Nhdr) + > > > > roundup(note->n_namesz, 4) + > > > > 617 note->n_descsz == notesz) > > > > 618 BCOPY((char *)note, buf, > > > > notesz); > > > > 619 } > > > > 620 } > > > > 621 > > > > 622 if (note->n_type != NT_PRSTATUS) { > > > > 623 error(WARNING, "invalid note (n_type != > > > > NT_PRSTATUS)\n"); > > > > 624 goto fail; > > > > 625 } > > > > > > > > Not sure how you want to handle that, probably just bail out the same way > > > > if note becomes NULL? > > > > > > If I add this to arm_get_crash_notes(), just after your new function: > > > > > > if (!note) { > > > error(WARNING, "cannot find NT_PRSTATUS note for cpu: %d\n", > > > i); > > > continue; > > > } > > > > > > I get this: > > > > > > $ ./crash /usr/dumps/ARM/vmcore.pae* /usr/dumps/ARM/vmlinux.pae.gz > > > > > > crash 7.1.4rc15 > > > Copyright (C) 2002-2014 Red Hat, Inc. > > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > > > Copyright (C) 1999-2006 Hewlett-Packard Co > > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > > Copyright (C) 2005, 2011 NEC Corporation > > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > This program is free software, covered by the GNU General Public License, > > > and you are welcome to change it and/or distribute copies of it under > > > certain conditions. Enter "help copying" to see the conditions. > > > This program has absolutely no warranty. Enter "help warranty" for > > > details. > > > > > > GNU gdb (GDB) 7.6 > > > Copyright (C) 2013 Free Software Foundation, Inc. > > > License GPLv3+: GNU GPL version 3 or later > > > <http://gnu.org/licenses/gpl.html> > > > This is free software: you are free to change and redistribute it. > > > There is NO WARRANTY, to the extent permitted by law. Type "show > > > copying" > > > and "show warranty" for details. > > > This GDB was configured as "--host=x86_64-unknown-linux-gnu > > > --target=arm-elf-linux"... > > > > > > WARNING: cannot find NT_PRSTATUS note for cpu: 1 > > > WARNING: cannot find NT_PRSTATUS note for cpu: 2 > > > WARNING: cannot find NT_PRSTATUS note for cpu: 3 > > > WARNING: cannot find NT_PRSTATUS note for cpu: 4 > > > KERNEL: /usr/dumps/ARM/vmlinux.pae.gz > > > DUMPFILE: /usr/dumps/ARM/vmcore.pae > > > CPUS: 5 [OFFLINE: 4] > > > DATE: Sun Jun 8 18:27:39 2014 > > > UPTIME: 00:03:22 > > > LOAD AVERAGE: 0.16, 0.16, 0.07 > > > TASKS: 51 > > > NODENAME: buildroot > > > RELEASE: 3.13.5 > > > VERSION: #3 SMP Mon Jun 9 05:58:39 CST 2014 > > > MACHINE: armv7l (unknown Mhz) > > > MEMORY: 256 MB > > > PANIC: "SysRq : Trigger a crash" > > > PID: 732 > > > COMMAND: "sh" > > > TASK: 8bcead00 [THREAD_INFO: 8ad32000] > > > CPU: 0 > > > STATE: TASK_RUNNING (SYSRQ) > > > > > > crash> bt -a > > > PID: 732 TASK: 8bcead00 CPU: 0 COMMAND: "sh" > > > #0 [<80265064>] (sysrq_handle_crash) from [<80265810>] > > > #1 [<80265810>] (__handle_sysrq) from [<80265928>] > > > #2 [<80265928>] (write_sysrq_trigger) from [<80112120>] > > > #3 [<80112120>] (proc_reg_write) from [<800c9840>] > > > #4 [<800c9840>] (vfs_write) from [<800c9be4>] > > > #5 [<800c9be4>] (sys_write) from [<8000e3e0>] > > > pc : [<76e9cfdc>] lr : [<0000f998>] psr: 600d0010 > > > sp : 7eab862c ip : 00000000 fp : 000a82a4 > > > r10: 00000020 r9 : 000a8294 r8 : 00000001 > > > r7 : 00000004 r6 : 000a9bf0 r5 : 00000001 r4 : 000a7d88 > > > r3 : 00000000 r2 : 00000002 r1 : 000a9bf0 r0 : 00000001 > > > Flags: nZCv IRQs on FIQs on Mode USER_32 ISA ARM > > > > > > PID: 0 TASK: 8bc561c0 CPU: 1 COMMAND: "swapper/1" > > > bt: WARNING: cannot determine starting stack frame for task 8bc561c0 > > > > > > PID: 0 TASK: 8bc56580 CPU: 2 COMMAND: "swapper/2" > > > bt: WARNING: cannot determine starting stack frame for task 8bc56580 > > > > > > PID: 0 TASK: 8bc56940 CPU: 3 COMMAND: "swapper/3" > > > bt: WARNING: cannot determine starting stack frame for task 8bc56940 > > > > > > PID: 0 TASK: 8bc56d00 CPU: 4 COMMAND: "swapper/4" > > > bt: WARNING: cannot determine starting stack frame for task 8bc56d00 > > > crash> > > > > > > Note that if I did "goto fail" instead of "continue", I lose the good > > > cpu 0 backtrace from the NT_PRSTATUS that your patch found, so doing it > > > this way is the best of both worlds. > > > > I agree that the 'if (!note) continue' that you added is a good idea to > > try and salvage this type of dump. It shouldn't happen with qemu generated > > dumps, but anything's possible when a kernel panics... > > > > Would you like me to spin a v4 with this condition added? Or, since it > > actually seems to be addressing a non-qemu-generated dump issue, then > > maybe you just want to submit it as a new patch on top of the qemu > > patch? > > No, I'm cleaning it up now -- there are a couple other gotcha's w/respect > to backtracing, since we're now going to keep the ms->panic_task_regs[cpus] > array intact, even if there's no note in some of them. I'm going to make > sure it's zero-filled in the not-found case, and I have to make sure that > it's recognized on both architectures. I'll post the the updated additions > to arm.c and arm64.c so that you can do a quick test on a legitimate > QEMU dump. > > Might be Monday, I'm not sure... Sounds good to me :-) Thanks, drew -- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility
