On 11/19/2015 12:36 AM, Nan Xiao wrote: > Hi David & Dave, > > Executing "crash" on a physical machine (not VirtualBox): > > # crash > > crash 7.1.3 > Copyright (C) 2002-2014 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for details. > > crash: /boot/xen-4.5.gz: original filename unknown > Use "-f /boot/xen-4.5.gz" on command line to prevent this message. > > WARNING: machine type mismatch: > crash utility: X86_64 > /var/tmp/xen-4.5.gz_VIOmfp: X86 > > crash: /boot/symtypes-3.12.49-6-default.gz: original filename unknown > Use "-f /boot/symtypes-3.12.49-6-default.gz" on command line to > prevent this message. > > crash: /boot/symvers-3.12.49-6-default.gz: original filename unknown > Use "-f /boot/symvers-3.12.49-6-default.gz" on command line to > prevent this message. > > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-unknown-linux-gnu"... > > crash: this kernel may be configured with CONFIG_STRICT_DEVMEM, which > renders /dev/mem unusable as a live memory source. > crash: trying /proc/kcore as an alternative to /dev/mem > > KERNEL: /boot/vmlinux-3.12.49-6-xen.gz > DEBUGINFO: /usr/lib/debug/boot/vmlinux-3.12.49-6-xen.debug > DUMPFILE: /proc/kcore > CPUS: 128 > DATE: Thu Nov 19 09:37:49 2015 > UPTIME: 00:34:57 > LOAD AVERAGE: 1.77, 1.21, 1.02 > TASKS: 1328 > NODENAME: dl980-5 > RELEASE: 3.12.49-6-xen > VERSION: #1 SMP Mon Oct 26 16:05:37 UTC 2015 (11560c3) > MACHINE: x86_64 (1995 Mhz) > MEMORY: 125.9 GB > PID: 39777 > COMMAND: "crash" > TASK: ffff881eacaaa100 [THREAD_INFO: ffff881e8ff46000] > CPU: 3 > STATE: TASK_RUNNING (ACTIVE) > > crash> > > I can see the crash will use "/proc/kcore" instead of "/dev/mem". So I > try the same thing on VirtualBox: > > # crash /boot/vmlinux-3.12.49-6-xen.gz /proc/kcore > > crash 7.1.3 > Copyright (C) 2002-2014 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for details. > > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-unknown-linux-gnu"... > > KERNEL: /boot/vmlinux-3.12.49-6-xen.gz > DEBUGINFO: /usr/lib/debug/boot/vmlinux-3.12.49-6-xen.debug > DUMPFILE: /proc/kcore > CPUS: 1 > DATE: Thu Nov 19 01:53:01 2015 > UPTIME: 05:42:13 > LOAD AVERAGE: 0.19, 0.06, 0.06 > TASKS: 239 > NODENAME: linux-6ev3 > RELEASE: 3.12.49-6-xen > VERSION: #1 SMP Mon Oct 26 16:05:37 UTC 2015 (11560c3) > MACHINE: x86_64 (2594 Mhz) > MEMORY: 855.2 MB > PID: 3106 > COMMAND: "crash" > TASK: ffff88002ec5c040 [THREAD_INFO: ffff88000b3e2000] > CPU: 0 > STATE: TASK_RUNNING (ACTIVE) > > crash> > > It seems OK now. > > So my questions are: > > (1) Is it OK to use "/proc/kcore" instead of "/dev/mem" as a workaround? > Is there any side-effect? As I read it, /proc/kcore is the kernel's virtual address space and /dev/mem is the system's physical address space. It probably isn't wise to debug the latter on any dom0 (whether nested in another virtualization or not) except in very acute cases. It may explain the problem you were having in the first place if VirtualBox affects whether /dev/mem is a real physical memory view or if it doesn't then whether VirtualBox itself affects those cases where, as I understand it, the kernel has constant physical addresses for some things. > (2) Execute "crash -d8" on physical machine will cause crash utility core dump. > Use gdb to debug it: > > # gdb /usr/bin/crash core-crash-11-0-0-40072-1447945769 > GNU gdb (GDB; SUSE Linux Enterprise 12) 7.9.1 > Copyright (C) 2015 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-suse-linux". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <http://bugs.opensuse.org/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /usr/bin/crash...Reading symbols from > /usr/lib/debug/usr/bin/crash.debug...done. > done. > [New LWP 40072] > Core was generated by `crash -d8'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0x00007f5119001fd0 in get_cie_encoding (cie=0x7f5119004cd8) at > ../../../libgcc/unwind-dw2-fde.c:272 > 272 ../../../libgcc/unwind-dw2-fde.c: No such file or directory. > (gdb) > (gdb) bt > #0 0x00007f5119001fd0 in get_cie_encoding (cie=0x7f5119004cd8) at > ../../../libgcc/unwind-dw2-fde.c:272 > #1 0x00007f5119002699 in get_fde_encoding (f=0x7f5119006050) at > ../../../libgcc/unwind-dw2-fde.c:319 > #2 _Unwind_IteratePhdrCallback (info=info@entry=0x7fff463c10e0, > size=size@entry=64, ptr=ptr@entry=0x7fff463c1160) > at ../../../libgcc/unwind-dw2-fde-dip.c:408 > #3 0x00007f51196a3f3c in __GI___dl_iterate_phdr > (callback=callback@entry=0x7f5119002270 <_Unwind_IteratePhdrCallback>, > data=data@entry=0x7fff463c1160) at dl-iteratephdr.c:76 > #4 0x00007f51190035c3 in _Unwind_Find_FDE (pc=0x7f5119001aa7 > <_Unwind_Backtrace+55>, bases=bases@entry=0x7fff463c1498) > at ../../../libgcc/unwind-dw2-fde-dip.c:459 > #5 0x00007f5118ffff86 in uw_frame_state_for > (context=context@entry=0x7fff463c13f0, fs=fs@entry=0x7fff463c1240) > at ../../../libgcc/unwind-dw2.c:1241 > #6 0x00007f51190011d0 in uw_init_context_1 > (context=context@entry=0x7fff463c13f0, > outer_cfa=outer_cfa@entry=0x7fff463c16a0, > outer_ra=0x7f511967bf46 <__GI___backtrace+86>) at > ../../../libgcc/unwind-dw2.c:1562 > #7 0x00007f5119001aa8 in _Unwind_Backtrace (trace=0x7f511967bdd0 > <backtrace_helper>, trace_argument=0x7fff463c16a0) > at ../../../libgcc/unwind.inc:283 > #8 0x00007f511967bf46 in __GI___backtrace > (array=array@entry=0x7fff463c1710, size=size@entry=4) at > ../sysdeps/x86_64/backtrace.c:109 > #9 0x000000000047add7 in __error (type=type@entry=1, > fmt=fmt@entry=0x853d38 "read(/dev/mem, %lx, %ld): %ld (%lx)\n") at > tools.c:52 > #10 0x0000000000490c91 in read_dev_mem (fd=4, bufptr=0x7fff463c1f28, > cnt=8, addr=0, paddr=1052672) at memory.c:2298 > #11 0x0000000000486398 in readmem (addr=1052672, > memtype=memtype@entry=4, buffer=buffer@entry=0x7fff463c1f28, > size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2198 > #12 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > #13 readmem (addr=1052672, memtype=memtype@entry=4, > buffer=buffer@entry=0x7fff463c1fb8, size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2209 > #14 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > #15 readmem (addr=1052672, memtype=memtype@entry=4, > buffer=buffer@entry=0x7fff463c2048, size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2209 > #16 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > #17 readmem (addr=1052672, memtype=memtype@entry=4, > buffer=buffer@entry=0x7fff463c20d8, size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2209 > #18 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > #19 readmem (addr=1052672, memtype=memtype@entry=4, > buffer=buffer@entry=0x7fff463c2168, size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2209 > #20 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > #21 readmem (addr=1052672, memtype=memtype@entry=4, > buffer=buffer@entry=0x7fff463c21f8, size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2209 > #22 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > #23 readmem (addr=1052672, memtype=memtype@entry=4, > buffer=buffer@entry=0x7fff463c2288, size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2209 > #24 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > > ...... > > Below is always: > readmem (addr=1052672, memtype=memtype@entry=4, > buffer=buffer@entry=0x7fff463c2288, size=size@entry=8, > type=type@entry=0x8563e1 "devmem_is_allowed - pfn 257", > error_handle=error_handle@entry=6) at memory.c:2209 > 0x000000000048704a in devmem_is_restricted () at memory.c:2414 > > Seems a dead-loop, but not sure. That reads like a bug in the code to decide to switch to /proc/kcore. It is at a test if /dev/mem is allowed for debugging by verifying this from the function comment: * On x86 and x86_64, only the first 256 pages of physical memory * are accessible: It's considered restricted if a sizeof(long) read from the start of physical page 255 (address 1,044,480) succeeds and a sizeof(long) read from page 257 fails (address 1052672). Note the nesting readmem()'s first arguments. Since there's no checking of the actual value read at those tests, their failure status is how the memory file is considered restricted or not. However,the actual method of implementation is like this simplified view: readmem(addr, PHYSADDR, buffer, size...) { /* Compute from arguments the memory file position to read from */ . . . /* Perform the read of the open file descriptor for the memory file, returns errno for the read */ switch(READMEM(fd, buffer, count...)) { . . . case READ_ERROR: if (PRINT_ERROR_MESSAGE) { if ((pc->flags & DEVMEM) && (kt->flags & PRE_KERNEL_INIT) && devmem_is_restricted() && switch_to_proc_kcore()) { return (readmem(addr, memtype, buffer, size...); } } } The devmem_is_restricted() contains two readmem() of it's own and there's no protection against it nesting on the switch() above that I can see. FWIW, I need to spend more time to commit to a solution. You have a support contract anyway, right? Like Dave said, SUSE is the main source of Xen features in crash and what's needed now isn't really worth being dragged out on the list. If you do have a support contract and could report the issue it would probably help me a bit. -- David. -- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility