Preliminary tests look good on x86_64, and for the most part ia64.
On ia64, some modules never load any .data, .data.rel, etc. sections,
and when that happens the "p" command for their data (d) symbols
doesn't work. But other modules work just fine? No
regression
though...
On s390x, we don't stand a prayer, because the modules themselves
don't seem to contain any debug info? It looks like this:
crash> mod -l
MODULE
NAME
SIZE OBJECT FILE
2081a800 dm_mirror
56184 (not loaded) [CONFIG_KALLSYMS]
20826b00 sd_mod
44584 (not loaded) [CONFIG_KALLSYMS]
2082f100 dasd_fba_mod
29696 (not loaded) [CONFIG_KALLSYMS]
2084c600 jbd
100688 (not loaded) [CONFIG_KALLSYMS]
20867900 dasd_mod
116892 (not loaded) [CONFIG_KALLSYMS]
20880600 dasd_eckd_mod
84224 (not loaded) [CONFIG_KALLSYMS]
2088d900 dm_zero
19456 (not loaded) [CONFIG_KALLSYMS]
208c0b00 ext3
200208 (not loaded) [CONFIG_KALLSYMS]
208e9600 scsi_transport_fc
67848 (not loaded) [CONFIG_KALLSYMS]
208f4e00 dm_snapshot
41288 (not loaded) [CONFIG_KALLSYMS]
2094b900 scsi_mod
233104 (not loaded) [CONFIG_KALLSYMS]
20966100 qdio
103768 (not loaded) [CONFIG_KALLSYMS]
2097f900 dm_mod
99544 (not loaded) [CONFIG_KALLSYMS]
20993c00 ccwgroup
73472 (not loaded) [CONFIG_KALLSYMS]
209a5300 cu3088
67080 (not loaded) [CONFIG_KALLSYMS]
209f5a00 zfcp
236832 (not loaded) [CONFIG_KALLSYMS]
20a4ed00 sg
151600 (not loaded) [CONFIG_KALLSYMS]
20bcbf00 autofs4
254472 (not loaded) [CONFIG_KALLSYMS]
20c67600 qeth
563600 (not loaded) [CONFIG_KALLSYMS]
20fb0000 sunrpc
1076664 (not loaded) [CONFIG_KALLSYMS]
212aa100 ipv6
2300920 (not loaded) [CONFIG_KALLSYMS]
crash> mod -s ext3
MODULE
NAME
SIZE OBJECT FILE
208c0b00 ext3
200208 /lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko
crash> p ext3_file_operations
ext3_file_operations = $1 = 0
crash> whatis ext3_file_operations
<data variable, no debug info> ext3_file_operations;
crash> sym ext3_file_operations
208bff88 (D) ext3_file_operations
crash>
The add-symbol-file line looks OK, although the address of
ext3_file_operations looks like it's in the .data.rel.ro_addr
section (?):
add-symbol-file request add-symbol-file /lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko
0x208953a8 -s .exit.text 0x208adfc4 -s .rodata 0x208ae000 -s .rodata.str1.2
0x208af1b8 -s __versions 0x208b1d60 -s .data.rel.ro 0x208bfea0 -s .data.rel
0x208c01f8 -s .data.rel.local 0x208c0740 -s .gnu.linkonce.this_module 0x208c0b00
-s .bss 0x208c4e00
add symbol table from file "/lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko"
at
.text_addr = 0x208953a8
.exit.text_addr = 0x208adfc4
.rodata_addr = 0x208ae000
.rodata.str1.2_addr
= 0x208af1b8
__versions_addr = 0x208b1d60
.data.rel.ro_addr =
0x208bfea0
.data.rel_addr = 0x208c01f8
.data.rel.local_addr
= 0x208c0740
.gnu.linkonce.this_module_addr
= 0x208c0b00
.bss_addr = 0x208c4e00
Anyway, the behavior is identical to what it's always been,
so there's no regression.
I'll keep tinkering as I get time...
Thanks,
Dave
Castor Fu wrote:
Finding the overrun wasn't actually that hard. It's also fixed in the current GDB tree.I've attached a patch which fixes the problem in symfile.c. The point of this patchis to fix loading kernel modules symbol information on 2.6 for those who have notbeen following this.Hopefully this will work on other platforms too....-castorFrom: crash-utility-bounces@xxxxxxxxxx [mailto:crash-utility-bounces@xxxxxxxxxx] On Behalf Of Castor Fu
Sent: Wednesday, January 17, 2007 7:49 AM
To: Discussion list for crash utility usage, maintenance and development; Discussion list for crash utility usage, maintenance and development
Subject: RE: test results of latest 4.0-3.16.sym.patch (ia64)
Hi Dave:
I reproduced the problem on an x86 system by creating a module with a bunch
of sections.I then found the following in gdb-6.1/gdb/symfile.c:add_symbol_table_command()
num_sec_opts = 16;
with additional code for xreallocing if it turned out to have too many
sections.This seems to be the code which is broken. I'm loath to figure out exactly
what it is... I'll put together a patch against 4.0-3.17 which jacks this
up, and probably print a warning if we exceed the count.Thanks for digging so far into this.
-castor
-----Original Message-----
From: crash-utility-bounces@xxxxxxxxxx on behalf of Dave Anderson
Sent: Thu 1/4/2007 8:22 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: test results of latest 4.0-3.16.sym.patch (ia64)
Hi Castor,
Another FYI re: the xrealloc() crash. The problem appears
to be specific to gdb.I captured the "add-symbol-file" command string and saved
it in an input file. Then I brought crash up and executed
the input file, which simply passes the suspect command line
directly to gdb, and it crashes on its own:crash> < /tmp/junk
crash> add-symbol-file /lib/modules/2.6.18-1.2767.el5/kernel/net/ipv6/ipv6.ko 0xa00000021ed605b0 -s .exit.text 0xa00000021edb49a0 -s .rodata 0xa00000021edbd4c8 -s __ksymtab_strings 0xa00000021edbdc08 -s __versions 0xa00000021edbdf98 -s .data 0xa00000021edd6a20 -s .data.rel.ro 0xa00000021edd6c00 -s __ksymtab_gpl 0xa00000021edd6df8 -s __kcrctab_gpl 0xa00000021edd6ed8 -s .data.rel 0xa00000021edd6f48 -s .data.rel.local 0xa00000021ee39940 -s .data.rel.ro.local 0xa00000021ee3a9c0 -s .data.read_mostly 0xa00000021ee3a9e0 -s __ksymtab 0xa00000021ee3aa60 -s __kcrctab 0xa00000021ee3ac30 -s .gnu.linkonce.this_module 0xa00000021ee3ad80 -s .sdata 0xa00000021ee5d730 -s .bss 0xa00000021ee5b000 -s .sbss 0xa00000021ee5e8b8
add_symbol_file_command: calling xrealloc w/argcnt: 49 arg: [0xa00000021ee5d730]...
*** glibc detected *** ./crash: realloc(): invalid next size: 0x6000000001921fe0 ***
======= Backtrace: =========
/lib/libc.so.6.1[0x20000000002f2a70]
/lib/libc.so.6.1(realloc-0x1cb0b0)[0x20000000002f5e20]
./crash(xmrealloc+0x1fffffffffee6e20)[0x40000000003a7d00]
./crash[0x40000000002ff500]
./crash[0x40000000004221e0]
./crash(cmd_func+0x1ffffffffff61610)[0x4000000000422500]
./crash(execute_command+0x1fffffffffee25f0)[0x40000000003a34f0]
./crash(gdb_command_funnel+0x1fffffffffe2feb0)[0x40000000002f0dc0]
./crash(gdb_interface+0x1fffffffffcd7590)[0x40000000001984b0]
./crash(gdb_pass_through+0x1fffffffffcd6cb0)[0x4000000000197be0]
./crash(cmd_gdb+0x2000000000151068)[0x400000000019bbc0]
./crash(exec_command+0x1fffffffffb99db0)[0x400000000005acf0]
./crash(exec_input_file+0x1fffffffffd86d40)[0x4000000000247c90]
./crash[0x400000000005b420]
./crash(exec_command+0x1fffffffffb99e50)[0x400000000005ad90]
./crash(main_loop+0x1fffffffffb9a2e0)[0x400000000005a8e0]
./crash(current_interp_command_loop+0x200000000001fd60)[0x40000000004e0cc0]
./crash[0x40000000003199c0]
./crash[0x400000000039f370]
./crash[0x40000000003a4260]
./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320]
./crash[0x400000000031a930]
./crash[0x400000000039f370]
./crash[0x40000000003a4260]
./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320]
./crash(gdb_main+0x1fffffffffe58960)[0x40000000003198e0]
./crash(gdb_main_entry+0x1fffffffffe589f0)[0x4000000000319980]
./crash(gdb_main_loop+0x1fffffffffcd54d0)[0x4000000000196470]
./crash(main+0x1fffffffffb99820)[0x400000000005a330]
/lib/libc.so.6.1(__libc_start_main-0x2818f0)[0x200000000023f6c0]
./crash(_start+0x1fffffffffb95240)[0x4000000000056200]
======= Memory map: ========
00000000-00004000 r--p 00000000 00:00 0
2000000000000000-2000000000038000 r-xp 00000000 fd:00 10256390 /lib/ld-2.5.so
2000000000044000-2000000000050000 rw-p 00034000 fd:00 10256390 /lib/ld-2.5.so
2000000000050000-2000000000114000 r-xp 00000000 fd:00 10256405 /lib/libm-2.5.so
2000000000114000-2000000000120000 ---p 000c4000 fd:00 10256405 /lib/libm-2.5.so
2000000000120000-2000000000124000 rw-p 000c0000 fd:00 10256405 /lib/libm-2.5.so
2000000000124000-20000000001b0000 r-xp 00000000 fd:00 10883077 /usr/lib/libncurses.so.5.5
20000000001b0000-20000000001bc000 ---p 0008c000 fd:00 10883077 /usr/lib/libncurses.so.5.5
20000000001bc000-20000000001cc000 rw-p 00088000 fd:00 10883077 /usr/lib/libncurses.so.5.5
20000000001cc000-20000000001d0000 rw-p 20000000001cc000 00:00 0
20000000001d0000-20000000001d8000 r-xp 00000000 fd:00 10256403 /lib/libdl-2.5.so
20000000001d8000-20000000001e4000 ---p 00008000 fd:00 10256403 /lib/libdl-2.5.so
20000000001e4000-20000000001e8000 rw-p 00004000 fd:00 10256403 /lib/libdl-2.5.so
20000000001e8000-200000000020c000 r-xp 00000000 fd:00 10882711 /usr/lib/libz.so.1.2.3
200000000020c000-2000000000218000 ---p 00024000 fd:00 10882711 /usr/lib/libz.so.1.2.3
2000000000218000-200000000021c000 rw-p 00020000 fd:00 10882711 /usr/lib/libz.so.1.2.3
200000000021c000-2000000000480000 r-xp 00000000 fd:00 10256397 /lib/libc-2.5.so
2000000000480000-200000000048c000 ---p 00264000 fd:00 10256397 /lib/libc-2.5.so
200000000048c000-2000000000498000 rw-p 00260000 fd:00 10256397 /lib/libc-2.5.so
2000000000498000-20000000004d8000 rw-p 2000000000498000 00:00 0
20000000004d8000-2000000003c1c000 r--p 00000000 fd:00 10882710 /usr/lib/locale/locale-archive
2000000003c1c000-2000000003c2c000 rw-p 2000000003c1c000 00:00 0
2000000003c38000-2000000003c44000 r-xp 00000000 fd:00 10256427 /lib/libthread_db-1.0.so
2000000003c44000-2000000003c50000 ---p 0000c000 fd:00 10256427 /lib/libthread_db-1.0.so
2000000003c50000-2000000003c54000 rw-p 00008000 fd:00 10256427 /lib/libthread_db-1.0.so
2000000003c54000-2000000003c58000 rw-p 2000000003c54000 00:00 0
2000000003c6c000-2000000003da0000 rw-p 2000000003c6c000 00:00 0
2000000003da0000-2000000003dbc000 r-xp 00000000 fd:00 10884674 /usr/lib/libunwind.so.7.0.0
2000000003dbc000-2000000003dc8000 ---p 0001c000 fd:00 10884674 /usr/lib/libunwind.so.7.0.0
2000000003dc8000-2000000003dcc000 rw-p 00018000 fd:00 10884674 /usr/lib/libunwind.so.7.0.0
2000000003dcc000-2000000003df0000 rw-p 2000000003dcc000 00:00 0
2000000003e00000-2000000003e08000 r--s 00000000 fd:00 10977539 /usr/lib/gconv/gconv-modules.cache
2000000003e08000-2000000003e18000 rw-p 2000000003e08000 00:00 0
2000000003e1c000-2000000006edc000 rw-p 2000000003e1c000 00:00 0
2000000006ee8000-2000000006f04000 r-xp 00000000 fd:00 10256386 /lib/libgcc_s-4.1.1-20061130.so.1
2000000006f04000-2000000006f10000 ---p 0001c000 fd:00 10256386 /lib/libgcc_s-4.1.1-20061130.so.1
2000000006f10000-2000000006f14000 rw-p 00018000 fd:00 10256386 /lib/libgcc_s-4.1.1-20061130.so.1
2000000006f14000-2000000006f24000 rw-p 2000000006f14000 00:00 0
2000000008000000-2000000008024000 rw-p 2000000008000000 00:00 0
2000000008024000-200000000c000000 ---p 2000000008024000 00:00 0
4000000000000000-40000000007e0000 r-xp 00000000 fd:00 9633909 /var/tmp/crash-4.0-3.16/crash
600000000000c000-600000000006c000 rw-p 007dc000 fd:00 9633909 /var/tmp/crash-4.0-3.16/crash
600000000006c000-6000000001fc0000 rw-p 600000000006c000 00:00 0 [heap]
60000fff7fffc000-60000fff80004000 rw-p 60000fff7fffc000 00:00 0
60000ffffe068000-60000ffffe0bc000 rw-p 60000ffffe068000 00:00 0 [stack]
a000000000000000-a000000000020000 ---p 00000000 00:00 0 [vdso]
Aborted
-- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility
-- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility
