On Fri, Sep 11, 2020 at 9:45 AM Luka Oreskovic
<luka.oreskovic@xxxxxxxxxx> wrote:
>
> Greetings everyone,
>
> I have been testing various BPF programs on the ARM32 architecture and
> have encountered a strange error.
>
> When trying to run a simple program that prints out the arguments of
> the open syscall,
> I found some strange behaviour with the pointer offsets when accessing
> the arguments:
> The output of llvm-objdump differed from the verifier error dump log.
> Notice the differences in lines 0 and 1. Why is the bytecode being
> altered at runtime?
>
> I attached the program, the llvm-objdump result and the verifier dump below.
>
> Best wishes,
> Luka Oreskovic
>
> BPF program
> --------------------------------------------
> #include "vmlinux.h"
> #include <bpf/bpf_helpers.h>
>
> SEC("tracepoint/syscalls/sys_enter_open")
> int tracepoint__syscalls__sys_enter_open(struct trace_event_raw_sys_enter* ctx)
> {
> const char *arg1 = (const char *)ctx->args[0];
> int arg2 = (int)ctx->args[1];
>
> bpf_printk("Open arg 1: %s\n", arg1);
> bpf_printk("Open arg 2: %d\n", arg2);
>
> return 0;
> }
>
> char LICENSE[] SEC("license") = "GPL";
>
>
> llvm-objdump of program
> --------------------------------------------
> Disassembly of section tracepoint/syscalls/sys_enter_open:
>
> 0000000000000000 tracepoint__syscalls__sys_enter_open:
> ; int arg2 = (int)ctx->args[1];
> 0: 79 16 18 00 00 00 00 00 r6 = *(u64 *)(r1 + 24)
> ; const char *arg1 = (const char *)ctx->args[0];
> 1: 79 13 10 00 00 00 00 00 r3 = *(u64 *)(r1 + 16)
> 2: 18 01 00 00 20 31 3a 20 00 00 00 00 25 73 0a 00 r1 =
> 2941353058775328 ll
> ; bpf_printk("Open arg 1: %s\n", arg1);
> 4: 7b 1a f8 ff 00 00 00 00 *(u64 *)(r10 - 8) = r1
> 5: 18 07 00 00 4f 70 65 6e 00 00 00 00 20 61 72 67 r7 =
> 7454127125170581583 ll
> 7: 7b 7a f0 ff 00 00 00 00 *(u64 *)(r10 - 16) = r7
> 8: bf a1 00 00 00 00 00 00 r1 = r10
> 9: 07 01 00 00 f0 ff ff ff r1 += -16
> 10: b7 02 00 00 10 00 00 00 r2 = 16
> 11: 85 00 00 00 06 00 00 00 call 6
> 12: 18 01 00 00 20 32 3a 20 00 00 00 00 25 64 0a 00 r1 =
> 2924860384358944 ll
> ; bpf_printk("Open arg 2: %d\n", arg2);
> 14: 7b 1a f8 ff 00 00 00 00 *(u64 *)(r10 - 8) = r1
> 15: 7b 7a f0 ff 00 00 00 00 *(u64 *)(r10 - 16) = r7
> 16: bf a1 00 00 00 00 00 00 r1 = r10
> 17: 07 01 00 00 f0 ff ff ff r1 += -16
> 18: b7 02 00 00 10 00 00 00 r2 = 16
> 19: bf 63 00 00 00 00 00 00 r3 = r6
> 20: 85 00 00 00 06 00 00 00 call 6
> ; return 0;
> 21: b7 00 00 00 00 00 00 00 r0 = 0
> 22: 95 00 00 00 00 00 00 00 exit
>
>
> verifier output when running program
> --------------------------------------------
> libbpf: -- BEGIN DUMP LOG ---
> libbpf:
> Unrecognized arg#0 type PTR
> ; int arg2 = (int)ctx->args[1];
> 0: (79) r6 = *(u64 *)(r1 +16)
> ; const char *arg1 = (const char *)ctx->args[0];
> 1: (79) r3 = *(u64 *)(r1 +12)
> invalid bpf_context access off=12 size=8
> processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0
> peak_states 0 mark_read 0
>
> libbpf: -- END LOG --
One suspect would be libbpf's CO-RE relocations. Can you send full
debug libbpf logs, it will have a full log of what libbpf adjusted.
Please also include the definition of struct trace_event_raw_sys_enter
from your vmlinux.h, as well as commit that your kernel was built from
(to check the original definition).
