Message: 1
Date: Fri, 8 Aug 2008 16:10:34 +0530
From: "Durga Prasad" <durgaprasadsharma@xxxxxxxxx>
Subject: Viewing module symbols using crash
To: crash-utility@xxxxxxxxxx
Message-ID:
<4295d5fd0808080340h3996b141oacb4eee4eb9ee00e@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
Hi,
[1]:
I need to debug a kernel module (a device driver).
I added the desired module to crash using mod -S <path to module>. The
driver is compiled with -g option.
I could not see the structures defined within the module.
Am I missing something?
System info:
KERNEL: vmlinux.sles10sp2.smp.x86_64
DUMPFILE: 2008-08-08-18:57/vmcore
CPUS: 4
RELEASE: 2.6.16.60-0.21-smp
VERSION: #1 SMP Tue May 6 12:41:02 UTC 2008
MACHINE: x86_64 (1995 Mhz)
MEMORY: 3.9 GB
[2]:
I could get into live kernel debugging using crash on RHEL 5 after
installing relevant kernel-debug packages. How do I do the same on SLES 10?
As of now, I have been compiling the kernel with CONFIG_DEBUG_INFO set. I
can use the resulting vmlinux for post crash analysis, but not for live
analysis. Any light on how to accomplish this would be appreciated.
Thanks
Durga Prasad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.redhat.com/archives/crash-utility/attachments/20080808/bd7e28cf/attachment.html
------------------------------
Message: 2
Date: Fri, 08 Aug 2008 09:32:09 -0400
From: Dave Anderson <anderson@xxxxxxxxxx>
Subject: Re: Viewing module symbols using crash
To: "Discussion list for crash utility usage, maintenance and
development" <crash-utility@xxxxxxxxxx>
Cc: Bernhard Walle <bwalle@xxxxxxx>
Message-ID: <489C4AD9.4010309@xxxxxxxxxx>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Durga Prasad wrote:
> Hi,
>
> [1]:
> I need to debug a kernel module (a device driver).
> I added the desired module to crash using mod -S <path to module>. The
> driver is compiled with -g option.
> I could not see the structures defined within the module.
>
> Am I missing something?
Accessing data in kernel modules can be spotty at best,
but the embedded gdb module should typically be able
to pick up structure definitions in modules.
For example, these two structures are defined in ./fs/ext3/xattr.c
like so:
struct ext3_xattr_info {
int name_index;
const char *name;
const void *value;
size_t value_len;
};
struct ext3_xattr_search {
struct ext3_xattr_entry *first;
void *base;
void *end;
struct ext3_xattr_entry *here;
int not_found;
};
Without the ext3 module debuginfo, crash (gdb) doesn't know what
they are:
# crash -s
crash> struct ext3_xattr_search
struct: invalid data structure reference: ext3_xattr_search
crash> struct ext3_xattr_info
struct: invalid data structure reference: ext3_xattr_info
crash>
Until after adding the module debuginfo data:
crash> mod -s ext3
MODULE NAME SIZE OBJECT FILE
ffffffff8806be00 ext3 168017 /lib/modules/2.6.18-102.el5.perfmon2/kernel/fs/ext3/ext3.ko
crash> struct ext3_xattr_search
struct ext3_xattr_search {
struct ext3_xattr_entry *first;
void *base;
void *end;
struct ext3_xattr_entry *here;
int not_found;
}
SIZE: 40
crash> struct ext3_xattr_info
struct ext3_xattr_info {
int name_index;
const char *name;
const void *value;
size_t value_len;
}
SIZE: 32
crash>
Then you may be able to "p" individual instances of the structures
if there are symbolic/static declaration of them. Otherwise you
can always cast the address of the structure to the structure type,
as in:
crash> struct ext3_xattr_info <address>
If you cannot get the structure declaration after using "mod -[sS]",
then you'll have to get primitive, and just dump the data at the
relevant address with "rd".
>
> System info:
> KERNEL: vmlinux.sles10sp2.smp.x86_64
> DUMPFILE: 2008-08-08-18:57/vmcore
> CPUS: 4
>
> RELEASE: 2.6.16.60-0.21-smp
> VERSION: #1 SMP Tue May 6 12:41:02 UTC 2008
> MACHINE: x86_64 (1995 Mhz)
> MEMORY: 3.9 GB
>
>
>
> [2]:
> I could get into live kernel debugging using crash on RHEL 5 after
> installing relevant kernel-debug packages. How do I do the same on SLES 10?
> As of now, I have been compiling the kernel with CONFIG_DEBUG_INFO set.
> I can use the resulting vmlinux for post crash analysis, but not for
> live analysis. Any light on how to accomplish this would be appreciated.
For SLES I'm guessing that you could use the post-built (CONFIG_DEBUG_INFO) kernel,
but also add the System.map file of the live kernel to the command line, as in:
$ crash /boot/System.map vmlinux-post-built-with-debuginfo
Crash will use the debuginfo data from the vmlinux file, but then go in
and back-patch all the vmlinux file's minimal_symbols with the correct
symbol values from the System.map file.
But Bernhard Walle (bwalle@xxxxxxx) can answer that part of your question.
He's active on this list, but I've added him to the cc: list to make
sure he sees this.
Dave Anderson
>
> Thanks
> Durga Prasad
>
Hi Dave,
I attempted to try your suggested commands, but it tends to fail on my machine:
libl041:/var/log/dump # crash -s vmlinux.sles10sp2.smp.x86_64 2008-08-08-23\:27/vmcore
crash> struct ext3_xattr_search
struct: invalid data structure reference: ext3_xattr_search
crash> mod -s ext3
MODULE NAME SIZE OBJECT FILE
ffffffff8856a500 ext3 153360 /lib/modules/2.6.16.60-0.21-smp/kernel/fs/ext3/ext3.ko
crash> struct ext3_xattr_search
struct: invalid data structure reference: ext3_xattr_search
crash> struct ext3_xattr_info
struct: invalid data structure reference: ext3_xattr_info
crash> sys
KERNEL: vmlinux.sles10sp2.smp.x86_64
DUMPFILE: 2008-08-08-23:27/vmcore
CPUS: 4
DATE: Fri Aug 8 23:20:09 2008
UPTIME: 01:31:48
LOAD AVERAGE: 2.48, 1.32, 0.56
TASKS: 157
NODENAME: libl041
RELEASE: 2.6.16.60-0.21-smp
VERSION: #1 SMP Tue May 6 12:41:02 UTC 2008
MACHINE: x86_64 (1995 Mhz)
MEMORY: 3.9 GB
PANIC: "Oops: 0000 [1] SMP " (check log for details)
crash version: 4.0-4.10 gdb version: 6.1
I tried the same after compiling and installing the latest crash version (crash version: 4.0-6.3 gdb version: 6.1) with no difference in results.
What would you suggest?
Thanks
Durga
I attempted to try your suggested commands, but it tends to fail on my machine:
libl041:/var/log/dump # crash -s vmlinux.sles10sp2.smp.x86_64 2008-08-08-23\:27/vmcore
crash> struct ext3_xattr_search
struct: invalid data structure reference: ext3_xattr_search
crash> mod -s ext3
MODULE NAME SIZE OBJECT FILE
ffffffff8856a500 ext3 153360 /lib/modules/2.6.16.60-0.21-smp/kernel/fs/ext3/ext3.ko
crash> struct ext3_xattr_search
struct: invalid data structure reference: ext3_xattr_search
crash> struct ext3_xattr_info
struct: invalid data structure reference: ext3_xattr_info
crash> sys
KERNEL: vmlinux.sles10sp2.smp.x86_64
DUMPFILE: 2008-08-08-23:27/vmcore
CPUS: 4
DATE: Fri Aug 8 23:20:09 2008
UPTIME: 01:31:48
LOAD AVERAGE: 2.48, 1.32, 0.56
TASKS: 157
NODENAME: libl041
RELEASE: 2.6.16.60-0.21-smp
VERSION: #1 SMP Tue May 6 12:41:02 UTC 2008
MACHINE: x86_64 (1995 Mhz)
MEMORY: 3.9 GB
PANIC: "Oops: 0000 [1] SMP " (check log for details)
crash version: 4.0-4.10 gdb version: 6.1
I tried the same after compiling and installing the latest crash version (crash version: 4.0-6.3 gdb version: 6.1) with no difference in results.
What would you suggest?
Thanks
Durga
-- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility
