Re: Throw read error on vmcore produced by ARM soc.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



2013/3/27 Dave Anderson <anderson@xxxxxxxxxx>:
>
>
> ----- Original Message -----
>> 2013/3/26 Dave Anderson <anderson@xxxxxxxxxx>:
>> >
>> >
>> > ----- Original Message -----
>> >> Hi, list.
>> >>
>> >> I use crash-utility to analyse crash dump core from ARM soc. When I
>> >> execute command below, I get the error "crash: read error: kernel
>> >> virtual address: c0c1e040  type: "first vmap_area va_start"". I also
>> >> test it by gdb. It works fine. The Linux kernel's version is
>> >> v3.0.8.
>> >>
>> >> hfli@pc1935:~/work/crash-utility$ ./crash vmlinux Vmcore
>> >>
>> >> crash 6.1.4
>> >> Copyright (C) 2002-2013  Red Hat, Inc.
>> >> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>> >> Copyright (C) 1999-2006  Hewlett-Packard Co
>> >> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>> >> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>> >> Copyright (C) 2005, 2011  NEC Corporation
>> >> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>> >> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>> >> This program is free software, covered by the GNU General Public License,
>> >> and you are welcome to change it and/or distribute copies of it under
>> >> certain conditions.  Enter "help copying" to see the conditions.
>> >> This program has absolutely no warranty.  Enter "help warranty" for
>> >> details.
>> >>
>> >> GNU gdb (GDB) 7.3.1
>> >> Copyright (C) 2011 Free Software Foundation, Inc.
>> >> License GPLv3+: GNU GPL version 3 or later
>> >> <http://gnu.org/licenses/gpl.html>
>> >> This is free software: you are free to change and redistribute it.
>> >> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> >> and "show warranty" for details.
>> >> This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-elf-linux"...
>> >>
>> >> crash: read error: kernel virtual address: c0c1e040  type: "first vmap_area va_start"
>> >>
>> >> Errors like the one above typically occur when the kernel and memory source
>> >> do not match.  These are the files being used:
>> >>
>> >>       KERNEL: vmlinux
>> >>     DUMPFILE: Vmcore
>> >
>> > You've answered your own question -- you should always see errors if the vmlinux
>> > kernel does not match the kernel crashed system.
>> >
>> > If you cannot find/access the original vmlinux file that was being run
>> > by the crashed kernel, then get the /boot/System.map file of the crashed
>> > kernel, and enter it on the command line:
>> Thanks for your reply.
>>
>> The vmlinux, include debug information, and crash kernel, is
>> cross-compile built and produced together. I couldn't understand why
>> crash throw this warning "kernel and source doesn't match".
>>
>> >
>> >  $ crash vmlinux Vmcore System.map
>> >
>> > The crash utility will replace all of the invalid symbol values from the
>> > "wrong" vmlinux file with their correct values from the System.map file.
>>
>>
>> A moment ago. I rebuilt the arm kernel source again. And took "echo c
>> > /proc/sysrq-trigger" command to trigger system panic. The status lists below.
>> hfli@pc1935:~/work/crash-utility$ ./crash vmlinux0327 Vmcore0327
>>
>> crash 6.1.4
>> Copyright (C) 2002-2013  Red Hat, Inc.
>> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>> Copyright (C) 1999-2006  Hewlett-Packard Co
>> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>> Copyright (C) 2005, 2011  NEC Corporation
>> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>> This program is free software, covered by the GNU General Public License,
>> and you are welcome to change it and/or distribute copies of it under
>> certain conditions.  Enter "help copying" to see the conditions.
>> This program has absolutely no warranty.  Enter "help warranty" for
>> details.
>>
>> GNU gdb (GDB) 7.3.1
>> Copyright (C) 2011 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-elf-linux"...
>>
>> please wait... (gathering kmem slab cache data)
>> crash: read error: kernel virtual address: c0c91840  type: "kmem_cache buffer"
>>
>> crash: unable to initialize kmem slab cache subsystem
>>
>>
>> WARNING: invalid note (n_type != NT_PRSTATUS)
>>
>> WARNING: could not retrieve crash_notes
>> please wait... (gathering task table data)
>> crash: cannot read pid_hash upid
>>
>> crash: cannot read pid_hash upid
>> please wait... (determining panic task)
>> WARNING: cannot get stackframe for task
>>       KERNEL: vmlinux0327
>>     DUMPFILE: Vmcore0327
>>         CPUS: 1
>>         DATE: Thu Jan  1 08:00:00 1970
>>       UPTIME: 00:00:00
>> LOAD AVERAGE: 0.00, 0.00, 0.00
>>        TASKS: 1
>>     NODENAME: 10.38.50.241
>>      RELEASE: 3.0.8-00010-gb7f16a3-dirty
>>      VERSION: #339 Wed Mar 27 10:39:43 CST 2013
>>      MACHINE: armv7l  (unknown Mhz)
>>       MEMORY: 19 MB
>>        PANIC: ""
>>          PID: 0
>>      COMMAND: "swapper"
>>         TASK: c02e0620  [THREAD_INFO: c02dc000]
>>          CPU: 0
>>        STATE: TASK_RUNNING (ACTIVE)
>>      WARNING: panic task not found
>>
>> crash>
>>
>>
>> It also didn't works so fine. Then I appended system.map, the output
>> result is also the same.
>
> OK, so then it's not clear to me why you're seeing those errors.
>
> Was the dumpfile created using kdump?  It almost looks like the dump
> was taken while the system was still running?  Have you *ever* created
> a dumpfile that resulted in an error-free crash session?

Yes, the dumpfile is created by kdump. The dump was taken by "echo c >
/proc/sysrq-trigger".

I will try another case by inserting a panic module tomorrow.
>
> Perhaps the ARM users on this list have seen this kind of thing?
>
> If you enter "crash -d8 ..." on the command line, you may get a better
> picture of what leads up to the errors shown above, and of most
> interest, the readmem() calls that generate the errors.  If you
> see a "crash: read error: ...", then that means that the dumpfile
> doesn't contain the physical page associated with the virtual
> address shown.  But it's not clear whether the address itself
> is legitimate, i.e., was it gathered from the wrong location.

Sounds reasonable.

>
>>
>> I try GDB to test it.
>> hfli@pc1935:~/work/crash-utility$ ./gdb-7.5/gdb/gdb vmlinux0327
>> Vmcore0327
>> GNU gdb (GDB) 7.5
>> Copyright (C) 2012 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>> copying"
>> and "show warranty" for details.
>> This GDB was configured as "--host=x86 --target=arm-linux-gnueabi".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from
>> /home/hfli/work/crash-utility/vmlinux0327...done.
>>
>> warning: exec file is newer than core file.
>
> Again, this bothers me -- why is it "newer" than the core file?
> Are you sure that they are *exactly* the same?

I am sure they are *exactly* the same. :-)

I'm not clear the internals of how to judge exec file and core file.

>
>> [New LWP 278]
>> #0  0xc0155f7c in sysrq_handle_crash (key=99) at
>> drivers/tty/sysrq.c:134
>> 134             *killer = 1;
>> (gdb) list
>> 129     {
>> 130             char *killer = NULL;
>> 131
>> 132             panic_on_oops = 1;      /* force panic */
>> 133             wmb();
>> 134             *killer = 1;
>> 135     }
>> 136     static struct sysrq_key_op sysrq_crash_op = {
>> 137             .handler        = sysrq_handle_crash,
>> 138             .help_msg       = "Crash",
>> (gdb)
>>
>> gdb also works fine.
>>
>
> It works fine for gdb in the very limited case above.  The crash utility
> is also "working fine" for a much more expansive access of the dumpfile.
> But if you tried to access the same locations in the dumpfile that the
> crash utility is doing during its initialization, then gdb would also
> fail.
>
> Let's take a simple example -- in your first email, you saw this error:
>
>  crash: read error: kernel virtual address: c0c1e040  type: "first vmap_area va_start"
>
> which came from here:
>
>         if (vt->flags & USE_VMAP_AREA) {
>                 get_symbol_data("vmap_area_list", sizeof(void *), &vmap_area);
>                 if (!vmap_area)
>                         return 0;
>                 if (!readmem(vmap_area - OFFSET(vmap_area_list) +
>                     OFFSET(vmap_area_va_start), KVADDR, &vmalloc_start,
>                     sizeof(void *), "first vmap_area va_start", RETURN_ON_ERROR))
>                         non_matching_kernel();
>
> If I look at a sample ARM dumpfile I have, I see this:
>
>  crash> p vmap_area_list
>  vmap_area_list = $8 = {
>    next = 0xc30d4d78,
>    prev = 0xc06702b8
>  }
>
> where the "next" pointer of 0xc30d4d78 above points to the "list" member
> of a vmap_area structure:
>
>  crash> struct vmap_area
>  struct vmap_area {
>      long unsigned int va_start;
>      long unsigned int va_end;
>      long unsigned int flags;
>      struct rb_node rb_node;
>      struct list_head list;         <== "next" points here
>      struct list_head purge_list;
>      void *private;
>      struct rcu_head rcu_head;
>  }
>  SIZE: 52
>  crash>
>
> And I can dump that vmap_area structure like this:
>
>  crash> struct -x vmap_area -l vmap_area.list 0xc30d4d78
>  struct vmap_area {
>    va_start = 0xbf000000,
>    va_end = 0xbf005000,
>    flags = 0x4,
>    rb_node = {
>      rb_parent_color = 0xc2ca076d,
>      rb_right = 0x0,
>      rb_left = 0x0
>    },
>    list = {
>      next = 0xc2ca0778,
>      prev = 0xc0411ed4
>    },
>    purge_list = {
>      next = 0x0,
>      prev = 0x0
>    },
>    private = 0xc3396860,
>    rcu_head = {
>      next = 0x0,
>      func = 0
>    }
>  }
>
> But your kernel found a "vmap_area_list.next" pointer of c0c1e040,
> but it was not accessible from the dumpfile.
>
> So either:
>
>  (1) the "vmap_area_list" symbol value was not correct, or
>  (2) the page containing the first vmap_area structure was
>      not included in the dumpfile.
>
> Problem (1) can happen if your crashed kernel doesn't match the
> vmlinux file, i.e., the symbol values don't match.  But if the
> "vmap_area_list" symbol was correct, then (2) mush have occurred,
> and that should never happen unless the dumpfile was corrupted or
> was created incorrectly.
>

Agree.

Thanks for your patience again.

For my case, the crashkernel cmdline of crash kernel is
crashkernel=20M@10M. When the capture kernel launch, the
elfcorehdr=0x1d00000, and the initialization of /proc/vmcore will fail
with WARN_ON(pfn_valid(pfn)) throwing.

The routine is vmcore_init->parse_crash_elf_headers->read_from_oldmem->copy_oldmem_page->ioremap->__arm_ioremap->arch_ioremap_caller->__arm_ioremap_caller->__arm_ioremap_pfn_caller->WARN_ON(pfn_valid(pfn)).

My temporary solution is comment the WARN_ON() to make /proc/vmcore work.

May my comment method corrupt the vmcore?

Thanks.

> Dave
>
> --
> Crash-utility mailing list
> Crash-utility@xxxxxxxxxx
> https://www.redhat.com/mailman/listinfo/crash-utility

--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility




[Index of Archives]     [Fedora Development]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]     [Fedora Tools]

 

Powered by Linux