Tried running crash on a running kernel... seems that 4.0-3.7 doesn't like my kernel. When I run crash 4.0-7.2 on a live system, it appears that it has no problems with vmalloc'd module memory.
crash 4.0-3.7
...
GNU gdb 6.1
...
This GDB was configured as "i686-pc-linux-gnu"...
crash: /boot/System.map-2.6.20-17.39-custom2 and /dev/mem do not match!
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Enter "crash -h" for details.
crash 4.0-7.2
...
GNU gdb 6.1
...
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux-2.6.20-17.39-custom2
DUMPFILE: /dev/mem
CPUS: 2
DATE: Wed Oct 1 16:31:39 2008
UPTIME: 04:57:53
LOAD AVERAGE: 0.10, 0.09, 0.09
TASKS: 95
NODENAME: ProCurve-TMS-zl-Module
RELEASE: 2.6.20-17.39-custom2
VERSION: #3 SMP Wed Sep 24 10:11:03 PDT 2008
MACHINE: i686 (2200 Mhz)
MEMORY: 5 GB
PID: 15801
COMMAND: "crash"
TASK: 47bd6030 [THREAD_INFO: 4a8a8000]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash>
Since that seems ok (and I don't encounter the error) I'll run crash with -d7 on the dump file to hopefully expose what is wrong with either the dump or with crash.
I've attached the output of crash with -d7... not sure how the mailing like handles file attachments, but if needed I can paste the text. (or if there is something specific I should look for let me know and I can paste just that section).
-Kevin
-----Original Message-----
From: crash-utility-bounces@xxxxxxxxxx [mailto:crash-utility-bounces@xxxxxxxxxx] On Behalf Of Dave Anderson
Sent: Wednesday, October 01, 2008 12:44 PM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Worth, Kevin wrote:
> Hello kexec and crash mailing lists,
>
>
>
> Sorry to spam whoever's code this ISN'T an issue with, but I really am
> unsure of whether is a kdump or a crash issue. I am running an Ubuntu
> 7.04 with a 2.6.20 kernel (includes Ubuntus patches- source at
> http://packages.ubuntu.com/feisty/linux-source-2.6.20 ) and a modified
> VMSPLIT/PAGE_OFFSET value (see bottom for details) on an i386 machine
> with 4GB of memory. At first I thought this could be an issue with
> makedumpfile stripping out things it shouldn't, but I've found that
> setting up my initrd script so that it simply performs "cp /proc/vmcore
> /var/crash/vmcore" results in the same issue.
>
>
>
> I've tried this with both crash 4.0-6.3 and 4.0-7.2 and get the same
> result. Unfortunately I'm locked at kernel 2.6.20 for other reasons, or
> else I would try that.
>
>
>
> If anyone can offer suggestions of what to try, please let me know. If
> this is something that has already been resolved elsewhere, sorry to
> waste time, and if someone can point me to what resolved it, perhaps I
> can look at backporting the fix myself. Thanks for your time.
>
>
>
> crash-4.0-7.2$ ./crash ~/vmcore ~/targetfiles/vmlinux-2.6.20-17.39-custom2
>
>
>
> crash 4.0-7.2
>
> <snip>Copyright notices...</snip>
>
> GNU gdb 6.1
>
> <snip>Copyright notices...</snip>
>
> This GDB was configured as "i686-pc-linux-gnu"...
>
>
>
> please wait... (gathering module symbol data)
>
> WARNING: cannot access vmalloc'd module memory
>
>
>
> KERNEL: /home/worthk/targetfiles/vmlinux-2.6.20-17.39-custom2
>
> DUMPFILE: /home/worthk/vmcore
>
> CPUS: 2
>
> DATE: Wed Oct 1 12:30:50 2008
>
> UPTIME: 00:35:11
>
> LOAD AVERAGE: 0.07, 0.09, 0.08
>
> TASKS: 94
>
> NODENAME: test-module
>
> RELEASE: 2.6.20-17.39-custom2
>
> VERSION: #3 SMP Wed Sep 24 10:11:03 PDT 2008
>
> MACHINE: i686 (2200 Mhz)
>
> MEMORY: 5 GB
>
> <6>SysRq : Trigger a crashdump"
>
> PID: 4304
>
> COMMAND: "bash"
>
> TASK: 5d7e9030 [THREAD_INFO: f4b70000]
>
> CPU: 0
>
> STATE: TASK_RUNNING (SYSRQ)
>
>
>
> crash> mod -s test
>
> mod: cannot access vmalloc'd module memory
>
>
>
>
>
> My kernel config is a bit outside the norm, in that the VMSPLIT value
> has been modified to give 3GB of memory the kernelspace and 1GB of
> memory to userspace. Below is a diff between the default Ubuntu
> "generic" config and mine:
To be honest with you I'm surprised it comes up at all...
If you do a "crash -d7 vmlinux vmcore", amongst the reams of
debug data you will see the readmem() that failed just prior
to the "WARNING: cannot access vmalloc'd module memory". And
that will probably be the very first access of a vmalloc'd
virtual memory address. Probably it's best to enter
"crash -d7 vmlinux vmcore > /tmp/junk", and then enter "q"
to silently kill the session.
For that matter, once you come up, I'm guessing that user
virtual address translation will fail as well. Come up
as you did above, do a "vm" command on the "bash" task,
and then a "vtop" on a user virtual address.
Like this example:
crash> vm
PID: 25479 TASK: f6f2aaa0 CPU: 3 COMMAND: "bash"
MM PGD RSS TOTAL_VM
f6e3d740 f745c980 1560k 4608k
VMA START END FLAGS FILE
f6c115f4 110000 112000 75 /lib/
f7212f94 112000 113000 100071 /lib/
f78cd0cc 113000 114000 100073 /lib/
f6954d84 584000 585000 8000075
f7241bcc 5b1000 5b4000 75 /lib/libtermcap.so.2.0.8.#prelink#.YYRDOu
f7212a14 5b4000 5b5000 100073 /lib/libtermcap.so.2.0.8.#prelink#.YYRDOu
f6e1a64c 61a000 623000 75 /lib/libnss_files-2.5.so
f73f738c 623000 624000 100071 /lib/libnss_files-2.5.so
f6eb79bc 624000 625000 100073 /lib/libnss_files-2.5.so
f7212f3c 719000 733000 875 /lib/
f721238c 733000 734000 100871 /lib/
f72e2b1c 734000 735000 100873 /lib/
f73f7964 ab5000 bf2000 75 /lib/
f6c11ee4 bf2000 bf4000 100071 /lib/
f73f7ee4 bf4000 bf5000 100073 /lib/
f73f7f3c bf5000 bf8000 100073
f721217c 8048000 80f5000 1875 /bin/
f6cab90c 80f5000 80fa000 101873 /bin/
f724143c 80fa000 80ff000 100073
f68cf7ac 8574000 85aa000 100073
f6d354ec b7d81000 b7f81000 71 /usr/lib/locale/locale-archive
f7594d84 b7f81000 b7f84000 100073
f6f24124 b7f84000 b7f85000 100073
f72e2d2c b7f85000 b7f8c000 d1 /usr/lib/gconv/gconv-modules.cache
f72418b4 bfa8a000 bfa9f000 100173
crash> vtop 584000
VIRTUAL PHYSICAL
584000 37cd6000
PAGE DIRECTORY: f745c980
PGD: f745c980 => 369e0001
PMD: 369e0010 => 11b7c1067
PTE: 11b7c1c20 => 37cd6025
PAGE: 37cd6000
PTE PHYSICAL FLAGS
37cd6025 37cd6000 (PRESENT|USER|ACCESSED)
VMA START END FLAGS FILE
f6954d84 584000 585000 8000075
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
c16f9ac0 37cd6000 0 0 48 80000004
crash>
Does the vtop command fall apart somewhere?
BTW, if you haven't done it already, you should also
take the dumpfile out of the picture, and just run crash
on the live system. If by some stretch of the imagination
*that* works, then you might have to point the finger back
at kdump operation.
In any case, at least you've got a situation where crash
can at least deal with unity-mapped addresses. With those
addresses it doesn't have to do any kind of page-table
walk-throughs.
I'm guessing that there's something in x86.c's x86_init() function
in the PRE_GDB section that's not correct for your setup.
There is support for Red Hat's older "hugemem" 4G/4G split
kernels, where both the kernel and user space have 4G
(over-lapping) virtual address regions, and so there may be
some confusion there with yours.
For starters, bring up the session as you did above,
and enter "help -v" and "help -m". They're debug
options that dump a couple internal crash data structures
which may shed some light.
Dave
--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility
Attachment:
crash.log
Description: crash.log
-- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility
