On 3/14/22 12:09 AM, Shung-Hsi Yu wrote:
On Wed, Mar 09, 2022 at 03:20:47PM -0800, Stephen Brennan wrote:
Hello everyone,
I've been recently learning about BTF with a keen interest in using it
as a fallback source of debug information. On the face of it, Linux
kernels these days have a lot of introspection information. BTF provides
information about types. kallsyms provides information about symbol
locations. ORC allows us to reliably unwind stack traces. So together,
these could enable a debugger (either postmortem, or live) to do a lot
without needing to read the (very large) DWARF debuginfo files. For
example, we could format backtraces with function names, we could
For backtraces with function names, you probably still need ksyms since
BTF won't encode address => symbol translation.
pretty-print global variables and data structures, etc. This is nice
This indeed is a potential use case.
We discussed this during adding per-cpu
global variables. Ultimately we just added per-cpu global variables
since we didn't have a use case or request for other global variables.
But I still would like to know beyond this whether you have other needs
which BPF may or may not help. It would be good to know since if
ultimately you still need dwarf, then it might be undesirable to
add general global variables to BTF.
given that depending on your distro, it might be tough to get debuginfo,
and it is quite large to download or install.
As I've worked toward this goal, I discovered that while the
BTF_KIND_VAR exists [1], the BTF included in the core kernel only has
declarations for percpu variables. This makes BTF much less useful for
this (admittedly odd) use case. Without a way to bind a name found in
kallsyms to its type, we can't interpret global variables. It looks like
the restriction for percpu-only variables is baked into the pahole BTF
encoder [2].
[1]: https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-kind-var
[2]: https://github.com/acmel/dwarves/blob/master/btf_encoder.c
I wonder what the BPF / BTF community's thoughts are on including more
of these global variable declarations? Perhaps behind a
CONFIG_DEBUG_INFO_BTF_ALL, like how kallsyms does it? I'm aware that
Currently on my local machine, the vmlinux BTF's size is 4.2MB and
adding 1MB would be a big increase. CONFIG_DEBUG_INFO_BTF_ALL is a good
idea. But we might be able to just add global variables without this
new config if we have strong use case.
each declaration costs at least 16 bytes of BTF records, plus the
strings and any necessary type data. The string cost could be mitigated
by allowing "name_off" to refer to the kallsyms offset for variable or
function declaration. But the additional records could cost around 1MiB
for common distribution configurations.
I know this isn't the designed use case for BTF, but I think it's very
exciting.
I've been wondering about the same (possibility of using BTF for postmortem
debugging without debuginfo), though not to the extend that you've
researched.
I find the idea exciting as well, and quite useful for distros where the
kernel package changes quite often that the debuginfo package may be long
gone by the time a crash dump for such kernel is captured.
I would love to use BTF (including global variables in BTF) for crash
dump. But I suspect we may still have some gaps. Maybe you can
explore a little bit more on this?
Shung-Hsi
Thanks for your attention!
Stephen
