Version 4 of this series is a rebase over latest 3.8-rc4+. Moreover, I
updated the mechanism that implements automatic symbol loading for new
modules. It was affected by the refactorings around finit_module.
While waiting for feedback who could imagine picking this up for merge,
I wrote a tiny tutorial, see below.
Here is the original series intro again:
This adds the infrastructure and first tools that make kernel debugging
through gdb more comfortable. Since 7.0, gdb supports python scripting.
And this opens the doors to automate steps like the tedious loading of
module symbols at the right address, resolving per-cpu variables or even
retrieving the current kernel log without resuming an stopped target.
Many of the helpers naturally depend on the layout of structures or
internal mechanics of the kernel. So the best place to maintain such
things, keeping them consistent with the corresponding kernel is, well,
the kernel itself.
While these scripts have been originally developed for debugging via
QEMU/KVM, I've now also added the required bits for KGDB. Works fine,
but as QEMU/KVM tends to outperform KGDB it remains the recommendation
- when available.
There are two architecture dependencies so far, one regarding per-cpu,
the other regarding thread_info calculation. None of them I was able to
test on a target, so I'm counting on review/testing by the corresponding
communities.
This series should be considered the foundation of much more kernel
state exploration helpers, e.g. around tasks, timers, locks, sockets -
I guess people will have even more ideas.
And this is a tutorial for the gdb extension using QEMU/KVM as target
platform:
o Set up a virtual Linux machine for KVM (see www.linux-kvm.org and
www.qemu.org for more details)
o Build the kernel with this series applied, enabling CONFIG_DEBUG_INFO
(but leave CONFIG_DEBUG_INFO_REDUCED off)
o Install that kernel on the guest
o Enable the gdb stub of QEMU/KVM, either
- at VM startup time by appending "-s" to the QEMU command line
or
- during runtime by issuing "gdbserver" from the QEMU monitor
console
o cd /path/to/linux-build
o Start gdb: gdb vmlinux
o Attach to the booted guest:
(gdb) target remote :1234
o Load module (and main kernel) symbols:
(gdb) lx-symbols
loading vmlinux
scanning for modules in /home/user/linux/build
loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
...
loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
o Set a breakpoint on some not yet loaded module function, e.g.:
(gdb) b btrfs_init_sysfs
Function "btrfs_init_sysfs" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (btrfs_init_sysfs) pending.
o Continue the target
o Load the module on the target and watch what happens:
loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
o Let's examine the current task a bit:
(gdb) p ().pid
= 4998
(gdb) p ().comm
= "modprobe\000\000\000\000\000\000\000"
o Dump the log buffer of target kernel:
(gdb) lx-dmesg
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
[ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
....
o Make use of the per-cpu helper for the current or a specified CPU:
(gdb) p ("runqueues").nr_running
= 1
(gdb) p ("runqueues", 2).nr_running
= 0
o And now we are digging deep into hrtimers using the container_of
helper:
(gdb) set = ("hrtimer_bases").clock_base[0].active.next
(gdb) p *(, "struct hrtimer", "node")
= {
node = {
node = {
__rb_parent_color = 18446612133355256072,
rb_right = 0x0 <irq_stack_union>,
rb_left = 0x0 <irq_stack_union>
},
expires = {
tv64 = 1835268000000
}
},
_softexpires = {
tv64 = 1835268000000
},
function = 0xffffffff81078232 <tick_sched_timer>,
base = 0xffff88003fd0d6f0,
state = 1,
start_pid = 0,
start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
start_comm = "swapper/2\000\000\000\000\000\000"
}
Hope this provided some ideas and inspirations on how the commands and
helper functions can support kernel development.
Enjoy,
Jan
PS: Also available via git://git.kiszka.org/linux.git queues/gdb-scripts
CC: "David S. Miller" <davem@xxxxxxxxxxxxx>
CC: Fenghua Yu <fenghua.yu@xxxxxxxxx>
CC: Kay Sievers <kay@xxxxxxxx>
CC: linux-ia64@xxxxxxxxxxxxxxx
CC: linux-kbuild@xxxxxxxxxxxxxxx
CC: Michal Marek <mmarek@xxxxxxx>
CC: sparclinux@xxxxxxxxxxxxxxx
CC: Tony Luck <tony.luck@xxxxxxxxx>
Jan Kiszka (13):
scripts/gdb: Add infrastructure
scripts/gdb: Add container_of helper and convenience function
scripts/gdb: Add lx-symbols command
scripts/gdb: Add get_target_endianness helper
scripts/gdb: Add read_u16/32/64 helpers
scripts/gdb: Add lx-dmesg command
scripts/gdb: Add task iteration helper
scripts/gdb: Add helper and convenience function to look up tasks
scripts/gdb: Add is_target_arch helper
scripts/gdb: Add internal helper and convenience function to retrieve
thread_info
scripts/gdb: Add get_gdbserver_type helper
scripts/gdb: Add internal helper and convenience function for per-cpu
lookup
scripts/gdb: Add lx_current convenience function
Makefile | 5 +-
scripts/Makefile | 3 +-
scripts/gdb/Makefile | 9 +++
scripts/gdb/dmesg.py | 63 ++++++++++++++++++
scripts/gdb/percpu.py | 76 ++++++++++++++++++++++
scripts/gdb/symbols.py | 153 ++++++++++++++++++++++++++++++++++++++++++++
scripts/gdb/task.py | 108 +++++++++++++++++++++++++++++++
scripts/gdb/utils.py | 137 +++++++++++++++++++++++++++++++++++++++
scripts/gdb/vmlinux-gdb.py | 28 ++++++++
9 files changed, 580 insertions(+), 2 deletions(-)
create mode 100644 scripts/gdb/Makefile
create mode 100644 scripts/gdb/dmesg.py
create mode 100644 scripts/gdb/percpu.py
create mode 100644 scripts/gdb/symbols.py
create mode 100644 scripts/gdb/task.py
create mode 100644 scripts/gdb/utils.py
create mode 100644 scripts/gdb/vmlinux-gdb.py
--
1.7.3.4
--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
