On 12/02/2025 00:49, Stephen Brennan wrote:
> Hello all,
>
> I was recently able to clear up some issues related to the interaction
> between kallsyms generation and BTF generation, which was interfering with
> using BTF with global variables in the kernel. I sent some patches[1] to
> the kernel to enable global_var. After further testing, I see that enabling
> global_var results in the following validation errors:
>
> BPF: #011 type_id=23691 offset=435904 size=96
> fbcon: Taking over console
> BPF:
> BPF: Invalid offset
> BPF:
>
> Thanks to Alan Maguire's quick debugging, we were able to find the root
> cause. It turned out that several btf_var_secinfo entries had the exact
> same offset (and name) as their predecessors. The reason was that these
> entries corresponded to some variables are declared as "__weak" and then
> overridden. For example:
>
> // kernel/bpf/helpers.c
> const struct bpf_func_proto bpf_get_current_task_proto __weak;
> // kernel/trace/bpf_trace.c
> const struct bpf_func_proto bpf_get_current_task_proto = {...
>
> Both declarations appear in the DWARF as variable declarations, but it
> seems that there is no way to find out which one of the declarations is
> "__weak". (I checked this via llvm-dwarfdump). Overall, in a simple kernel
> configuration, I found 47 btf_var_secinfo which had duplicated offsets. In
> each case, both secinfos referred to distinct VARs, which had identical
> names and types. All were due to the "__weak" symbol. We need to eliminate
> these duplicates in order for the BTF to be validated by the kernel.
>
> This patch series does the deduplication of the VAR and SECINFO in pahole,
> by collecting the lists of variables for each ELF section, and then
> outputting them all once the list is sorted by offset and duplicates are
> identified. The libbpf btf__dedup() function does not deduplicate DATASEC
> or VAR. It would probably be possible to implement this there, and I'm open
> to feedback or suggestions regarding this. I implemented it in pahole
> because I'm most familiar with that code, and because it seems to me like
> it's reasonable for libbpf to expect that the input variable information is
> already deduplicated.
>
I've been thinking about whether core BTF deduplication should handle
this case - I'll try and lay out the process and maybe we can think
about whether it's better to solve in core dedup or within pahole.
The deduplication of VARs should be straightforward - they are
considered reference types and since in cases like this they share a
name and refer to the same type, the reference type deduplication could
be extended to cover them. However, the DATASEC references to such
variables are a bit trickier.
As seen above, the kernel currently disallows DATASEC btf_var_secinfo
references that are overlapping, i.e. if I have
[123520] DATASEC '.data..percpu' size=229632 vlen=392
type_id=7708 offset=0 size=48 (VAR 'fixed_percpu_data')
type_id=5559 offset=4096 size=4096 (VAR 'cpu_debug_store')
type_id=7060 offset=8192 size=16384 (VAR 'irq_stack_backing_store')
..the kernel enforces that irq_stack_backing_store starts at >= the
offset of the previous var (cpu_debug_store) + its size (4096+4096 in
this case). This is why the kernel rejects BTF with multiple instances
of the same variable, since they overlap.
So if we consider the case of deduplicating variables; before dedup we
would have something like this in the DATASEC
type_id=8188 offset=256 size=48 (VAR 'foo')
type_id=9190 offset=256 size=48 (VAR 'foo')
...and after dedup + remapping it would be:
type_id=8188 offset=256 size=48 (VAR 'foo')
type_id=8188 offset=256 size=48 (VAR 'foo')
This still violates the overlap check. So there are a few options here:
- change the kernel to relax overlap check when multiple references have
same type id/offset/size; i.e. they refer to the same variable
- have pahole weed out such occurrences
- something else?
Ideally we don't want to have to resize DATASECs with such duplicate
entries as that would add more complexity to dedup.
Anyway I'd be interested to hear what others think about whether solving
this in BTF dedup itself or pahole makes most sense. Thanks!
Alan
> I've gone ahead and tested this by building & booting a kernel with these
> changes, and the kernel patch series at [1]. The result exhibited no BPF
> varidation errors, and the drgn BTF branch[2] is working perfectly with it!
>
> Thanks,
> Stephen
>
> [1]: https://lore.kernel.org/bpf/20250207012045.2129841-1-stephen.s.brennan@xxxxxxxxxx/
> [2]: https://github.com/brenns10/drgn/commits/btf_2024
>
> Stephen Brennan (3):
> btf_encoder: move btf_encoder__add_decl_tag()
> btf_encoder: postpone VARs until encoding DATASEC
> btf_encoder: don't encode duplicate variables
>
> btf_encoder.c | 234 ++++++++++++++++++++++++++++++++++++--------------
> 1 file changed, 168 insertions(+), 66 deletions(-)
>
