Hello all,
Currently, pahole has the "var" feature which emits a single DATASEC for
the ".data..percpu" section, and emits only the variables from that
section. I work on the drgn debugger [1] and have been working specifically
on features that allow users to debug their kernel without necessarily
having DWARF debuginfo [2]. To that end, I'd like to see *all* global
variables included in the BTF, not just percpu variables. But this woudln't
just be useful for debuggers: libbpf already supports resolution of global
variables with __ksym annotations, but these have to be declared as void.
For example, in tools/bpf/bpftool/skeleton/pid_iter.bpf.c we have:
extern const void bpf_map_fops __ksym;
With global variable information, declarations like these would be able to
use the actial variable types, for example:
extern const struct file_operations bpf_map_fops __ksym;
To that end, this patch series adds a pahole feature "global_var", which
includes global variable type information in the BTF. Most of the important
code comes in patch 4. The major difficulty is in deciding which DWARF
variables should be emitted. The current pahole implementation considers
any variable with a static memory address a global, but that includes
static local variables, so I've added a flag to indicate when variables are
declared at the top level of a CU. There are some other useless kernel
variables which are generated by macros, etc. I filter these by name, which
isn't ideal, but works well.
To test and measure the size impact, I've built v6.11-rc7 on x86_64 with a
defconfig, and enabled the necessary BTF support. For one build, I used
pahole built from master, and for the other, I used these patches on top.
Here is a comparison of total vmlinux and module BTF size for old and new.
Before After
vmlinux BTF size 0x5f4a03 (5.95 MiB) 0x77e948 (7.49 MiB)
increase: 1.54 MiB (25.8%)
#modules 10 10
module BTF size 0x20c5 (8.19 KiB) 0x3235 (12.55 KiB)
increase: 4.36 KiB (53.2%)
The module size increases more compared to the vmlinux, because each
module's BTF doesn't define many types, so the variables are larger as a
percentage of the total. These are also rather small modules (whichever few
were enabled for the build of an x86_64 defconfig), I'd expect larger
modules to have a smaller relative increase, because they would define more
of their own types.
To sanity check the output, I looked at the "DATASEC" generated for the
percpu section, and verified that it was nearly identical. The only
difference (besides the type IDs) was the addition of the fixed_percpu_data
variable, which had been omitted by pahole's master branch:
--- percpu_datasec_master.txt 2024-09-12 00:04:37.255240578 +0000
+++ percpu_datasec_global_var.txt 2024-09-12 00:04:24.214980821 +0000
@@ -1,4 +1,5 @@
-DATASEC '.data..percpu' size=199384 vlen=302
+DATASEC '.data..percpu' size=199384 vlen=303
+offset=0 size=48 (VAR 'fixed_percpu_data')
offset=4096 size=4096 (VAR 'cpu_debug_store')
offset=8192 size=16384 (VAR 'irq_stack_backing_store')
offset=24576 size=20480 (VAR 'cpu_tss_rw')
Finally, you can find below a summary of the vmlinux DATASEC outputs which
were added by this patch in this build scenario.
DATASEC '.rodata' size=2910928 vlen=6514
DATASEC '__init_rodata' size=48 vlen=1
DATASEC '__param' size=21640 vlen=537
DATASEC '__modver' size=1008 vlen=14
DATASEC '.data' size=2953985 vlen=16451
DATASEC '.orc_header' size=20 vlen=1
DATASEC '.vvar' size=656 vlen=2
DATASEC '.data..percpu' size=199384 vlen=303
DATASEC '.init.data' size=690040 vlen=4830
DATASEC '.x86_cpu_dev.init' size=40 vlen=5
DATASEC '.apicdrivers' size=16 vlen=1
DATASEC '.data_nosave' size=4 vlen=1
DATASEC '.bss' size=693060 vlen=2532
DATASEC '.brk' size=196608 vlen=2
I know the overheads are non-negligible, but I think there will be
considerable value in having the global variables present. Thanks for
reading this and I look forward to discussing!
[1]: https://github.com/osandov/drgn
[2]: https://github.com/osandov/drgn/issues/176
PS: Previous iterations were sent in 2022:
v1 https://lore.kernel.org/dwarves/20220826184911.168442-1-stephen.s.brennan@xxxxxxxxxx/
v2 https://lore.kernel.org/dwarves/20221104231103.752040-1-stephen.s.brennan@xxxxxxxxxx/
Difference from these: I've gotten a much better understanding of DWARF,
BTF, and the pahole code since then. In particular, the older patches did
not include DATASEC for anything other than percpu variables. Also, since
those older patches, I've dramatically improved the variable filtering,
including using DW_AT_artificial, name filtering, and getting rid of static
local variables.
Stephen Brennan (4):
dutil: return ELF section name when looked up by index
dwarf_loader: add "artificial" and "top_level" variable flags
btf_encoder: cache all ELF section info
btf_encoder: add global_var feature to encode globals
btf_encoder.c | 371 ++++++++++++++++++++++-----------------------
btf_encoder.h | 8 +
dutil.c | 9 +-
dutil.h | 2 +-
dwarf_loader.c | 12 +-
dwarves.h | 3 +
man-pages/pahole.1 | 8 +-
pahole.c | 11 +-
8 files changed, 223 insertions(+), 201 deletions(-)
--
2.43.5
