Re: Crash issue when loading vmcore

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



----- "Paul Cahier" <pkc@xxxxxxxxxxxx> wrote:

> Hello,
> 
> I have finished setting up kdump and kexec today, recompiling my kernel 
> to add everything needed in there.
> I have triggered a kernel panic by echo c>/proc/sysrq-trigger, and found 
> that the vmcore dump was indeed there after all was done.
> 
> However I can not get any traces out of that crash dump(short version,
> long version at the end of the email):
> 
> crash /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930
> [...]
> crash: read error: kernel virtual address: c148a9a0  type: "kernel_config_data"
> WARNING: cannot read kernel_config_data
> crash: read error: kernel virtual address: c1487e28  type: "cpu_possible_mask"

The virtual addresses for "kernel_config_data" and "cpu_possible_mask" are
strange (too high?) -- I'll continue the analysis at the end of your "d7" 
output below...
 
> If I try crash --minimal things do load but I'm stuck with the minimal
> error set that's not very helpful.
> All I'm looking at is getting a full trace of the kernel panic.
> 
> 
> - Paul-Kenji Cahier
> 
> 
> PS, the full version:
> crash -d7 /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930
> 
> crash 5.0.6
> Copyright (C) 2002-2010  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for
> details.
> 
> vmcore_data:
>                    flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
>                     ndfd: 3
>                      ofp: b77344c0
>              header_size: 1860
>     num_pt_load_segments: 9
>       pt_load_segment[0]:
>              file_offset: 744
>               phys_start: 0
>                 phys_end: a0000
>                zero_fill: 0
>       pt_load_segment[1]:
>              file_offset: a0744
>               phys_start: 100000
>                 phys_end: 1000000
>                zero_fill: 0
>       pt_load_segment[2]:
>              file_offset: fa0744
>               phys_start: 5000000
>                 phys_end: 38000000
>                zero_fill: 0
>       pt_load_segment[3]:
>              file_offset: 33fa0744
>               phys_start: 38000000
>                 phys_end: 3e5ff000
>                zero_fill: 0
>       pt_load_segment[4]:
>              file_offset: 3a59f744
>               phys_start: 3e6c6000
>                 phys_end: 3f594000
>                zero_fill: 0
>       pt_load_segment[5]:
>              file_offset: 3b46d744
>               phys_start: 3f59c000
>                 phys_end: 3f62a000
>                zero_fill: 0
>       pt_load_segment[6]:
>              file_offset: 3b4fb744
>               phys_start: 3f62e000
>                 phys_end: 3f6a9000
>                zero_fill: 0
>       pt_load_segment[7]:
>              file_offset: 3b576744
>               phys_start: 3f6e9000
>                 phys_end: 3f6ed000
>                zero_fill: 0
>       pt_load_segment[8]:
>              file_offset: 3b57a744
>               phys_start: 3f6ff000
>                 phys_end: 3f700000
>                zero_fill: 0
>               elf_header: 85368c0
>                    elf32: 85368c0
>                  notes32: 85368f4
>                   load32: 8536914
>                    elf64: 0
>                  notes64: 0
>                   load64: 0
>              nt_prstatus: 8536a34
>              nt_prpsinfo: 0
>            nt_taskstruct: 0
>              task_struct: 0
>                page_size: 0
>             switch_stack: 0
>           xen_kdump_data: (unused)
>         num_prstatus_notes: 2
>                 vmcoreinfo: 0
>            size_vmcoreinfo: 0
>         nt_prstatus_percpu:
>          08536a34 08536ad8
> 
> Elf32_Ehdr:
>                  e_ident: \177ELF
>        e_ident[EI_CLASS]: 1 (ELFCLASS32)
>         e_ident[EI_DATA]: 1 (ELFDATA2LSB)
>      e_ident[EI_VERSION]: 1 (EV_CURRENT)
>        e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
>   e_ident[EI_ABIVERSION]: 0
>                   e_type: 4 (ET_CORE)
>                e_machine: 3 (EM_386)
>                e_version: 1 (EV_CURRENT)
>                  e_entry: 0
>                  e_phoff: 34
>                  e_shoff: 0
>                  e_flags: 0
>                 e_ehsize: 34
>              e_phentsize: 20
>                  e_phnum: a
>              e_shentsize: 0
>                  e_shnum: 0
>               e_shstrndx: 0
> Elf32_Phdr:
>                   p_type: 4 (PT_NOTE)
>                 p_offset: 372 (174)
>                  p_vaddr: 0
>                  p_paddr: 0
>                 p_filesz: 1488 (5d0)
>                  p_memsz: 1488 (5d0)
>                  p_flags: 0 ()
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 1860 (744)
>                  p_vaddr: c0000000
>                  p_paddr: 0
>                 p_filesz: 655360 (a0000)
>                  p_memsz: 655360 (a0000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 657220 (a0744)
>                  p_vaddr: c0100000
>                  p_paddr: 100000
>                 p_filesz: 15728640 (f00000)
>                  p_memsz: 15728640 (f00000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 16385860 (fa0744)
>                  p_vaddr: c5000000
>                  p_paddr: 5000000
>                 p_filesz: 855638016 (33000000)
>                  p_memsz: 855638016 (33000000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 872023876 (33fa0744)
>                  p_vaddr: ffffffff
>                  p_paddr: 38000000
>                 p_filesz: 106950656 (65ff000)
>                  p_memsz: 106950656 (65ff000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 978974532 (3a59f744)
>                  p_vaddr: ffffffff
>                  p_paddr: 3e6c6000
>                 p_filesz: 15523840 (ece000)
>                  p_memsz: 15523840 (ece000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 994498372 (3b46d744)
>                  p_vaddr: ffffffff
>                  p_paddr: 3f59c000
>                 p_filesz: 581632 (8e000)
>                  p_memsz: 581632 (8e000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 995080004 (3b4fb744)
>                  p_vaddr: ffffffff
>                  p_paddr: 3f62e000
>                 p_filesz: 503808 (7b000)
>                  p_memsz: 503808 (7b000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 995583812 (3b576744)
>                  p_vaddr: ffffffff
>                  p_paddr: 3f6e9000
>                 p_filesz: 16384 (4000)
>                  p_memsz: 16384 (4000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Phdr:
>                   p_type: 1 (PT_LOAD)
>                 p_offset: 995600196 (3b57a744)
>                  p_vaddr: ffffffff
>                  p_paddr: 3f6ff000
>                 p_filesz: 4096 (1000)
>                  p_memsz: 4096 (1000)
>                  p_flags: 7 (PF_X|PF_W|PF_R)
>                  p_align: 0
> Elf32_Nhdr:
>                 n_namesz: 5 ("CORE")
>                 n_descsz: 144
>                   n_type: 1 (NT_PRSTATUS)
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 00000000 00000401
>                           c06fef00 00000001 0000f004 0000f008
>                           00000000 c06e3ecc 00000282 00000282
>                           00000024 c06e3fa4 00000068 00000000
> Elf32_Nhdr:
>                 n_namesz: 5 ("CORE")
>                 n_descsz: 144
>                   n_type: 1 (NT_PRSTATUS)
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 00000dbd 00000000
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 00000000 00000000
>                           00000000 00000000 c0712420 00007e7e
>                           00000000 00000063 00000000 f0639f0c
>                           00000063 0000007b 0000007b 000000d8
>                           00000033 ffffffff c03145b2 00000060
>                           00010086 f0639f0c 00000068 00000000
> Elf32_Nhdr:
>                 n_namesz: 11 ("VMCOREINFO")
>                 n_descsz: 1134
>                   n_type: 0 (unused)
>                           OSRELEASE=2.6.35.3-saber
>                           PAGESIZE=4096
>                           SYMBOL(init_uts_ns)=c06f9120
>                           SYMBOL(node_online_map)=c0730644
>                           SYMBOL(swapper_pg_dir)=c06e4000
>                           SYMBOL(_stext)=c0101000
>                           SYMBOL(vmlist)=c07d3540
>                           SYMBOL(mem_map)=c07d3500
>                           SYMBOL(contig_page_data)=c072ce80
>                           SIZE(page)=32
>                           SIZE(pglist_data)=4224
>                           SIZE(zone)=1024
>                           SIZE(free_area)=44
>                           SIZE(list_head)=8
>                           SIZE(nodemask_t)=4
>                           OFFSET(page.flags)=0
>                           OFFSET(page._count)=4
>                           OFFSET(page.mapping)=16
>                           OFFSET(page.lru)=24
>                           OFFSET(pglist_data.node_zones)=0
>                           OFFSET(pglist_data.nr_zones)=4140
>                           OFFSET(pglist_data.node_mem_map)=4144
>                           OFFSET(pglist_data.node_start_pfn)=4148
>                           OFFSET(pglist_data.node_spanned_pages)=4156
>                           OFFSET(pglist_data.node_id)=4160
>                           OFFSET(zone.free_area)=40
>                           OFFSET(zone.vm_stat)=728
>                           OFFSET(zone.spanned_pages)=916
>                           OFFSET(free_area.free_list)=0
>                           OFFSET(list_head.next)=0
>                           OFFSET(list_head.prev)=4
>                           OFFSET(vm_struct.addr)=4
>                           LENGTH(zone.free_area)=11
>                           SYMBOL(log_buf)=c06fc83c
>                           SYMBOL(log_end)=c07bb7ec
>                           SYMBOL(log_buf_len)=c06fc838
>                           SYMBOL(logged_chars)=c07c38a0
>                           LENGTH(free_area.free_list)=5
>                           NUMBER(NR_FREE_PAGES)=0
>                           NUMBER(PG_lru)=5
>                           NUMBER(PG_private)=11
>                           NUMBER(PG_swapcache)=16
>                           CONFIG_X86_PAE=y
>                           CRASHTIME=1282584565
> cannot determine relocation value: not a live system
> gdb /usr/src/linux-2.6.35.3/vmlinux
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
> 
> <readmem: c148a9a0, KVADDR, "kernel_config_data", 32768, (ROE), 8bad3d8>
> crash: read error: kernel virtual address: c148a9a0  type: "kernel_config_data"
> WARNING: cannot read kernel_config_data
> <readmem: c1487e28, KVADDR, "cpu_possible_mask", 4, (FOE), bfed4bbc>
> crash: read error: kernel virtual address: c1487e28  type: "cpu_possible_mask"

The read error with for the "kernel_config_data" symbol at c148a9a0 and (which returns on
error -- that's what ROE means), and then the "cpu_possible_mask" symbol at c1487e28 (which
cause the session to fault or bail out -- FOE), mean that -- after translating those virtual
addresses to physical addresses by stripping off the c0000000 unity-map identifier -- those 
physical addresses (at 148a9a0 and 1487e28 respectively) were not found in the dumpfile.
And that's because the ELF header of the vmcore does not show a PT_LOAD segment
that contains those physical addresses.

But as I mentioned before, the virtual addresses seem to be too high for
static kernel data symbols.  If you run --minimal, does the "sym" command
show "cpu_possible_mask" at that address?  I don't have anything later than
a 2.6.34 x86 dumpfile to use as a reference, but the symbol is much lower
in value in that kernel:

  crash> sym cpu_possible_mask
  c07ffa28 (R) cpu_possible_mask  
  crash>

And if I dump all of the symbols from within a --minimal session with that
dumpfile, I see this, where the "_end" of the static kernel virtual memory
is at c0c77000:

  crash> sym -l
  ... [ cut ] ...
  c0b50ffc (b) netlbl_unlhsh_lock
  c0b51000 (b) klist_remove_lock
  c0b51004 (B) __bss_stop
  c0b52000 (b) .brk
  c0b52000 (B) __brk_base
  c0b62000 (b) .brk.pagetables
  c0c67000 (b) .brk.dmi_alloc
  c0c77000 (B) __brk_limit
  c0c77000 (A) _end
  crash>

And if you look at the "VMCOREINFO" data above in your dump for items that are
kernel symbol values, they make sense, i.e., 

>                           SYMBOL(node_online_map)=c0730644
>                           SYMBOL(swapper_pg_dir)=c06e4000
>                           SYMBOL(_stext)=c0101000
>                           SYMBOL(vmlist)=c07d3540
>                           SYMBOL(mem_map)=c07d3500
>                           SYMBOL(contig_page_data)=c072ce80

If you run a --minimal session, what do you see when you run the
two commands that I show above?  (i.e., "sym cpu_possible_mask" & the output
of the tail end of "sym -l")

Dave




But for starters, if you run the --minimal session and then execute the

--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility


[Index of Archives]     [Fedora Development]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]     [Fedora Tools]

 

Powered by Linux