[Crash-utility] crash read error: kernel virtual address / vmcore address mismatch ?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, May 8, 2013 at 6:59 PM, Dave Anderson <anderson at redhat.com> wrote:
>
>
>
> ----- Original Message -----
>> On Wed, May 8, 2013 at 11:23 AM, Anand Raj Manickam <anandrm at gmail.com>
>> wrote:
>> >> Hi ,
>> >> Sorry about re posting this as i did not find solution ...
>> >>
>> >> I m facing a issue where on
>> >>
>> >> #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore
>> >
>> > Do yourself a favor and copy /proc/vmcore to somewhere on disk.  Then
>> > reboot the system into the original kernel, and run crash on the saved
>> > vmcore.  I've never seen crash run on a /proc/vmcore file from the
>> > secondary kernel.
>> >
>> > Also, after rebooting the the original kernel, please confirm that
>> > crash runs OK on the live system.
>> >
>> >
>> >> crash 6.1.6
>> >> Copyright (C) 2002-2013  Red Hat, Inc.
>> >> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>> >> Copyright (C) 1999-2006  Hewlett-Packard Co
>> >> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>> >> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>> >> Copyright (C) 2005, 2011  NEC Corporation
>> >> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>> >> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>> >> This program is free software, covered by the GNU General Public License,
>> >> and you are welcome to change it and/or distribute copies of it under
>> >> certain conditions.  Enter "help copying" to see the conditions.
>> >> This program has absolutely no warranty.  Enter "help warranty" for
>> >> details.
>> >>
>> >> GNU gdb (GDB) 7.3.1
>> >> Copyright (C) 2011 Free Software Foundation, Inc.
>> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> >> This is free software: you are free to change and redistribute it.
>> >> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> >> and "show warranty" for details.
>> >> This GDB was configured as "i686-pc-linux-gnu"...
>> >>
>> >> crash: read error: kernel virtual address: c127a9c8  type: "cpu_possible_mask"
>> >
>> > Please show the full output of "crash -d8 vmlinux vmcore".
>>
>> After rebooting to the primary / live kernel
>>
>>    I have attached the output for "crash -d8 vmlinux /data/vmcore" in d8.txt
>
> Looking at the ELF header data in the debug output, your configuration
> does require special handling.  X86 kernels are typically not configured
> with CONFIG_PHYSICAL_START larger than CONFIG_PHYSICAL_ALIGN, because
> when that is done, the vmlinux unity-mapped symbol values do not match
> their relocated values when the kernel is loaded.  This entry in the
> crash.changelog describes the situation:
>
> http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
>
> 4.0-4.5  - Addresses FC7/upstream x86 kernels that have been configured such
>            that the vmlinux symbol values do not match their relocated values
>            when loaded.  If CONFIG_PHYSICAL_START contains a value that is
>            greater then CONFIG_PHYSICAL_ALIGN, then this mismatch occurs.
>            Since the crash utility and its embedded gdb have always expected
>            that the compiled-in kernel symbol addresses are "real", the virtual
>            to physical translation fails, leading to an initialization-time
>            failure with the message: "crash: vmlinux and /dev/crash do not
>            match!" (/dev/mem or the dumpfile name may replace /dev/crash).
>            To deal with this issue, there are several alternatives:
>
>             1) Configure the kernel with CONFIG_PHYSICAL_START less than
>                or equal to CONFIG_PHYSICAL_ALIGN.  Having done that, there
>                is no problem; the resultant vmlinux file will be loaded at
>                the address for which it was compiled, which has always
>                been the case.
>             2) Since /proc/kallsyms uses the same format as a System.map file,
>                and since it reflects the relocated symbol addresses, it
>                can be placed on the crash command line as if it were
>                a System.map file.  (Note that the System.map file created
>                by these relocated kernels contains the same "wrong" symbol
>                values as the vmlinux file from which it was created.)
>             3) On a live system that has /proc/kallsyms (i.e., the kernel was
>                configured with CONFIG_KALLSYMS), this version of the crash
>                utility will replace/patch the vmlinux symbol values with those
>                seen in /proc/kallsyms.  The relocation value will be displayed
>                as a WARNING message during initialization.
>             4) On a dumpfile, the relocation will not be performed automatically
>                as on a live system.  It will require the addition of the
>                /proc/kallsyms on the command line, or if run on a different
>                host, a copy of the crashed system's /proc/kallsyms may be
>                used.
>             5) Alternatively on a dumpfile, a new command line option has been
>                created to specify the relocation amount.  For example, if a
>                kernel was configured with a CONFIG_PHYSICAL_START value of 16MB
>                and a CONFIG_PHYSICAL_ALIGN of 4MB, that results in a relocation
>                of 12MB.  To specify that, enter "crash --reloc=12m ..." on the
>                command line.  (Recall that if crash is run on the live system,
>                a WARNING message will specify the relocation amount.)
>
>            Using /proc/kallsyms or a --reloc=[size] as a command line argument
>            is similar to using a System.map file, in that it results in the loss
>            of the use of line number debug data.  (anderson at redhat.com)
>
>            ...
>
>>> My current config for
>>>
>>> CONFIG_PHYSICAL_START=0x1000000
>>> CONFIG_RELOCATABLE=y
>>> CONFIG_PHYSICAL_ALIGN=0x400000
>>>
>>> The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel.
>>> On the System Kernel it starts from 0x400000 .On the Debug kernel it
>>> starts from 0x1000000.
>
> So your configuration exactly matches paragraph 5) above, you should
> be able to come up OK by entering either:
>

>  $ crash vmlinux vmcore --reloc=12m

This did the trick :-)

Great !! and Thanks Dave.

>
> or
>
>  $ crash vmlinux vmcore /proc/kallsyms
>
> where the /proc/kallsyms is from the *primary* kernel, so option #2
> will *not* work when running from the secondary kernel.
>
> You didn't show any output of a live crash session (while running
> on the primary kernel), but I believe it should work without using
> the /proc/kallsyms as an argument or using --reloc=12m option.  There
> was a further fix for these configurations for running on live systems:
>
> http://people.redhat.com/anderson/crash.changelog.html#5_1_4
>
> 5.1.4    - Fix for RT kernels in which the schedule() function has become a
>            ... [ cut] ...
>
>          - Fix for running against live x86 kernels that were configured with
>            CONFIG_PHYSICAL_START containing a value that is greater than its
>            CONFIG_PHYSICAL_ALIGN value, and where the first symbol listed by
>            /proc/kallsyms is not "_text".  Without the patch, the crash session
>            fails during invocation with the error message "crash: vmlinux and
>            /dev/mem do not match!" (or "/dev/crash" if applicable).  As a work-
>            around, "/proc/kallsyms" can be entered on the command line, or the
>            "--reloc=<size>" option could be used, but the fix obviates that
>            requirement for live systems.  It should be noted that dumpfiles of
>            kernels configured that way still do require that "/proc/kallsyms",
>            or a copy of it, or alternatively the "--reloc=<size>" option, to
>            be entered on the command line, as detailed in this changelog entry:
>            http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
>            (anderson at redhat.com)
>
> So on the live system on the primary kernel, I believe this should just work:
>
>  $ crash vmlinux
>
> Normally you don't even need to put the vmlinux file on the command line for
> running on the live system because it will be "found" automatically unless
> you've put it in some non-standard location.
>
> Dave
>



[Index of Archives]     [LM Sensors]     [Linux Sound]     [ALSA Users]     [ALSA Devel]     [Linux Audio Users]     [Linux Media]     [Kernel]     [Gimp]     [Yosemite News]     [Linux Media]

  Powered by Linux