On 4/15/20 7:23 PM, David Ahern wrote:
On 4/15/20 1:27 PM, Yonghong Song wrote:
As there are some discussions regarding to the kernel interface/steps to
create file/anonymous dumpers, I think it will be beneficial for
discussion with this work in progress.
Motivation:
The current way to dump kernel data structures mostly:
1. /proc system
2. various specific tools like "ss" which requires kernel support.
3. drgn
The dropback for the first two is that whenever you want to dump more, you
need change the kernel. For example, Martin wants to dump socket local
If kernel support is needed for bpfdump of kernel data structures, you
are not really solving the kernel support problem. i.e., to dump
ipv4_route's you need to modify the relevant proc show function.
Yes, as mentioned two paragraphs below. kernel change is required.
The tradeoff is that this is a one-time investment. Once kernel change
is in place, printing new fields (in most cases except new fields
which need additional locks etc.) no need for kernel change any more.
storage with "ss". Kernel change is needed for it to work ([1]).
This is also the direct motivation for this work.
drgn ([2]) solves this proble nicely and no kernel change is not needed.
But since drgn is not able to verify the validity of a particular pointer value,
it might present the wrong results in rare cases.
In this patch set, we introduce bpf based dumping. Initial kernel changes are
still needed, but a data structure change will not require kernel changes
any more. bpf program itself is used to adapt to new data structure
changes. This will give certain flexibility with guaranteed correctness.
Here, kernel seq_ops is used to facilitate dumping, similar to current
/proc and many other lossless kernel dumping facilities.
User Interfaces:
1. A new mount file system, bpfdump at /sys/kernel/bpfdump is introduced.
Different from /sys/fs/bpf, this is a single user mount. Mount command
can be:
mount -t bpfdump bpfdump /sys/kernel/bpfdump
2. Kernel bpf dumpable data structures are represented as directories
under /sys/kernel/bpfdump, e.g.,
/sys/kernel/bpfdump/ipv6_route/
/sys/kernel/bpfdump/netlink/
The names of bpfdump fs entries do not match actual data structure names
- e.g., there is no ipv6_route struct. On the one hand that is a good
thing since structure names can change, but that also means a mapping is
needed between the dumper filesystem entries and what you get for context.
Yes, the later bpftool patch implements a new command to dump such
information.
$ bpftool dumper show target
target prog_ctx_type
task bpfdump__task
task/file bpfdump__task_file
bpf_map bpfdump__bpf_map
ipv6_route bpfdump__ipv6_route
netlink bpfdump__netlink
in vmlinux.h generated by vmlinux BTF, we have
struct bpf_dump_meta {
struct seq_file *seq;
u64 session_id;
u64 seq_num;
};
struct bpfdump__ipv6_route {
struct bpf_dump_meta *meta;
struct fib6_info *rt;
};
Here, bpfdump__ipv6_route is the bpf program context type.
User can based on this to write the bpf program.
Further, what is the expectation in terms of stable API for these fs
entries? Entries in the context can change. Data structure names can
change. Entries in the structs can change. All of that breaks the idea
of stable programs that are compiled once and run for all future
releases. When structs change, those programs will break - and
structures will change.
Yes, the API (ctx) we presented to bpf program is indeed unstable.
CO-RE should help to certain extend but if some fields are gone, e.g.,
bpf program will need to be rewritten for that particular kernel
version, or kernel bpfdump infrastructure can be enhanced to
change its ctx structure to have more information to the program
for that kernel version. In summary, I agree with you that this is
an unstable API similar to other tracing program
since it accesses kernel internal data structures.
What does bpfdumper provide that you can not do with a tracepoint on a
relevant function and then putting a program on the tracepoint? ie., why
not just put a tracepoint in the relevant dump functions.
In my very beginning to explore bpfdump, kprobe to "show" function is
one of options. But quickly we realized that we actually do not want
to just piggyback on "show" function, but want to replace it with
bpf. This will be useful in following different use cases:
1. first catable dumper file, similar to /proc/net/ipv6_route,
we want /sys/kernel/bpfdump/ipv6_route/my_dumper and you can cat
to get it.
Using kprobe when you are doing `cat /proc/net/ipv6_route`
is complicated. You probably need an application which
runs through `cat /proc/net/ipv6_route` and discard its output,
and at the same time gets the result from bpf program
(filtered by pid since somebody may run
`cat /proc/net/ipv6_route` at the same time. You may use
perf ring_buffer to send the result back to the application.
note that perf ring buffer may lose records for whatever
reason and seq_ops are implemented not to lose records
by built-in retries.
Using kprobe approach above is complicated and for each dumper
you need an application. We would like it to be just catable
with minimum user overhead to create such a dumper.
2. second, anonymous dumper, kprobe/tracepoint will incur
original overhead of seq_printf per object. but user may
be only interested in a very small portion of information.
In such cases, bpf program directly doing filtering in
the kernel can potentially speed up a lot if there are a lot of
records to traverse.
3. for data structures which do not have catable dumpers
for example task, hopefully, as demonstrated in this patch set,
kernel implementation and writing a bpf program are not
too hard. This especially enables people to do in-kernel
filtering which is the strength of the bpf.
