On Sun, Jun 12, 2022 at 7:11 AM Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote: > > Hi Andrii, > > Sorry for waiting. > > On Mon, 23 May 2022 22:57:48 -0700 > Andrii Nakryiko <andrii@xxxxxxxxxx> wrote: > > > Hi Masami, > > > > We've got reports about partially corrupt stack traces when being captured from > > uretprobes. Trying the simplest repro seems to confirm that something is not > > quite right here. I'll try to debug it a bit more this week, but I was hoping > > for you to take a look as well, if you get a chance. > > Ah, uprobes/uretprobes will be handled by Oleg. But I'll try to explain it > as far as I know. Oleg, please correct me if I'm wrong. > > > > > Simple repro built on top of BPF selftests. > > > > $ sudo ./test_progs -a uprobe_autoattach -v > > ... > > FN ADDR 0x55fde0 - 0x55fdef > > UPROBE SZ 40 (CNT 5) URETPROBE SZ 40 (CNT 5) > > UPROBE 0x55fde0 URETPROBE 0x55ffd4 > > UPROBE 0x584653 URETPROBE 0x584653 > > UPROBE 0x585cc9 URETPROBE 0x585cc9 > > UPROBE 0x7fa9a31eaca3 URETPROBE 0x7fa9a31eaca3 > > UPROBE 0x5541f689495641d7 URETPROBE 0x5541f689495641d7 > > ... > > #203 uprobe_autoattach:OK > > Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED > > > > There seem to be two distinct problems. > > > > 1. Last two entries for both uprobe and uretprobe stacks are not user-space > > addressed (0x7fa9a31eaca3) and the very last one doesn't even look like a valid > > address (0x5541f689495641d7). > > Hmm, what user-space stack unwinder are you using? I guess it comes > from the user-space stack unwinder routine. > > > > > 2. Looking at first entry for UPROBE vs URETPROBE, you can see that uprobe > > one's is correct and points exactly to the beginning of autoattach_trigger_func > > (0x55fde0) as expected, but uretprobe entry (0x55ffd4) is way out of > > autoattach_trigger_func (which is just 15 bytes long and ends at 0x55fdef). > > Using addr2line it shows that it points to: > > > > 0x000000000055ffd4: test_uprobe_autoattach at /data/users/andriin/linux/tools/testing/selftests/bpf/prog_tests/uprobe_autoattach.c:33 > > > > Which is a valid function and location to which autoattach_trigger_func() > > should return (see objdump snippet below), but from uretprobe I'd imagine that > > we are going to get address within traced user function (that is 0x55fde0 - > > 0x55fdef range), not the return address in a parent function. > > No, it is expected behavior. Since the "return probe" probes the function > returning, the first entry of the stack address is the address where the > function is returned (IOW, the IP address when the probe hit). Not the > address of the return instruction. As long as it uses the stack to hook > the return of the function, it's impossible in principle. > (if it decodes the function to find the return instruction and replace it > with a software breakpoint, it may be possible. But this will not work > if the function does tail call etc.) > > > > > 55ffc4: 89 83 3c 08 00 00 mov %eax,0x83c(%rbx) > > 55ffca: 8b 45 e8 mov -0x18(%rbp),%eax > > 55ffcd: 89 c7 mov %eax,%edi > > 55ffcf: e8 0c fe ff ff call 55fde0 <autoattach_trigger_func> > > --> 55ffd4: 89 45 a8 mov %eax,-0x58(%rbp) > > 55ffd7: ba ef fd 55 00 mov $0x55fdef,%edx > > > > Both issues above seem unexpected, can you please see if I have some wrong > > assumptions here? > > So, #1 maybe we need to look into the stack unwinder (bpf_get_stack()?) > Does this happen if you unwind user-stack from kprobe event? > Oleg, would you know anything about this issue? > I suspect that this is due to no-frame-pointer libc, so I haven't dug deeper. > And #2 is the expected behavior. > Ok, I see, thanks for confirming! > Thank you, > > > > > Thanks in advance for taking a look! > > > > Cc: Jiri Olsa <jolsa@xxxxxxxxxx> > > Cc: Riham Selim <rihams@xxxxxx> > > Signed-off-by: Andrii Nakryiko <andrii@xxxxxxxxxx> > > --- > > tools/testing/selftests/bpf/Makefile | 3 ++- > > .../selftests/bpf/prog_tests/uprobe_autoattach.c | 14 ++++++++++++++ > > .../selftests/bpf/progs/test_uprobe_autoattach.c | 11 +++++++++++ > > 3 files changed, 27 insertions(+), 1 deletion(-) > > [...] > > -- > Masami Hiramatsu (Google) <mhiramat@xxxxxxxxxx>
