On Thu, Apr 30, 2020 at 7:44 PM Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote:
>
> On Tue, 28 Apr 2020 09:19:47 -0300
> Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
>
> > Em Tue, Apr 28, 2020 at 12:47:53PM +0200, Eelco Chaudron escreveu:
> > > On 28 Apr 2020, at 6:04, Alexei Starovoitov wrote:
> > > > On Fri, Apr 24, 2020 at 02:29:56PM +0200, Eelco Chaudron wrote:
> >
> > > > > > But in reality I think few kprobes in the prog will be enough to
> > > > > > debug the program and XDP prog may still process millions of
> > > > > > packets because your kprobe could be in error path and the user
> > > > > > may want to capture only specific things when it triggers.
> >
> > > > > > kprobe bpf prog will execute in such case and it can capture
> > > > > > necessary state from xdp prog, from packet or from maps that xdp
> > > > > > prog is using.
> >
> > > > > > Some sort of bpf-gdb would be needed in user space. Obviously
> > > > > > people shouldn't be writing such kprob-bpf progs that debug
> > > > > > other bpf progs by hand. bpf-gdb should be able to generate them
> > > > > > automatically.
> >
> > > > > See my opening comment. What you're describing here is more when
> > > > > the right developer has access to the specific system. But this
> > > > > might not even be possible in some environments.
> >
> > > > All I'm saying that kprobe is a way to trace kernel.
> > > > The same facility should be used to trace bpf progs.
> >
> > > perf doesn’t support tracing bpf programs, do you know of any tools that
> > > can, or you have any examples that would do this?
> >
> > I'm discussing with Yonghong and Masami what would be needed for 'perf
> > probe' to be able to add kprobes to BPF jitted areas in addition to
> > vmlinux and modules.
>
> At a grance, at first we need a debuginfo which maps the source code and
> BPF binaries. We also need to get a map from the kernel indicating
> which instructions the bpf code was jited to.
> Are there any such information?
it's already there. Try 'bpftool prog dump jited id N'
It will show something like this:
; data = ({typeof(errors.leaf) *leaf =
bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &type_key); if (!leaf) {
bpf_map_update_elem_(bpf_pseudo_fd(1, -11), &type_key, &zero,
BPF_NOEXIST); leaf = bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &t;
81d: movabs $0xffff8881a0679000,%rdi
; return bpf_map_lookup_elem((void *)map, key);
827: mov %rbx,%rsi
82a: callq 0xffffffffe0f7f448
82f: test %rax,%rax
832: je 0x0000000000000838
834: add $0x40,%rax
; if (!data)
838: test %rax,%rax
83b: je 0x0000000000000846
83d: mov $0x1,%edi
; lock_xadd(data, 1);
842: lock add %edi,0x0(%rax)
> Also, I would like to know the target BPF (XDP) is running in kprobes
> context or not. BPF tracer sometimes use the kprobes to hook the event
> and run in the kprobe (INT3) context. That will be need more work to
> probe it.
> For the BPF code which just runs in tracepoint context, it will be easy
> to probe it. (we may need to break a limitation of notrace, which we
> already has a kconfig)
yeah. this mechanism won't be able to debug bpf progs that are
attached to kprobes via int3. But that is rare case.
Most kprobe+bpf are for function entry and adding int3 to jited bpf code
will work just like for normal kernel functions.
