On Thu, Oct 06, 2022 at 11:54:13AM +0200, Jiri Olsa wrote:
> On Thu, Oct 06, 2022 at 04:34:46PM +0900, Akihiro HARAI wrote:
> > Depending on distribution/kernel/syscall combination, BTF entry for
> > `struct pt_regs *regs` parameter differs.
> >
> > For example, Amazon Linux 2 with kernel-5.15 package enabled has a FWD
> > entry for `__x64_sys_recvmsg` function:
> >
> > ```
> > $ uname -a
> > Linux ip-10-1-1-66.ap-northeast-1.compute.internal
> > 5.15.43-20.123.amzn2.x86_64 #1 SMP Fri May 27 00:28:44 UTC 2022 x86_64
> > x86_64 x86_64 GNU/Linux
> >
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > ...
> > [15439] FWD 'pt_regs' fwd_kind=struct
> > [15440] CONST '(anon)' type_id=15439
> > [15441] PTR '(anon)' type_id=15440
> > [15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
> > '__unused' type_id=15441
> > ...
> > [15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
> > ...
> > ```
> >
> > while Ubuntu 20.04 LTS with newer kernel has a STRUCT entry for the
> > same function:
> >
> > ```
> > $ uname -a
> > Linux xxx-XPS-13-9300 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun
> > 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> > ...
> > [226] STRUCT 'pt_regs' size=168 vlen=21
> > 'r15' type_id=1 bits_offset=0
> > 'r14' type_id=1 bits_offset=64
> > 'r13' type_id=1 bits_offset=128
> > 'r12' type_id=1 bits_offset=192
> > 'bp' type_id=1 bits_offset=256
> > 'bx' type_id=1 bits_offset=320
> > 'r11' type_id=1 bits_offset=384
> > 'r10' type_id=1 bits_offset=448
> > 'r9' type_id=1 bits_offset=512
> > 'r8' type_id=1 bits_offset=576
> > 'ax' type_id=1 bits_offset=640
> > 'cx' type_id=1 bits_offset=704
> > 'dx' type_id=1 bits_offset=768
> > 'si' type_id=1 bits_offset=832
> > 'di' type_id=1 bits_offset=896
> > 'orig_ax' type_id=1 bits_offset=960
> > 'ip' type_id=1 bits_offset=1024
> > 'cs' type_id=1 bits_offset=1088
> > 'flags' type_id=1 bits_offset=1152
> > 'sp' type_id=1 bits_offset=1216
> > 'ss' type_id=1 bits_offset=1280
> > ...
> > [5183] CONST '(anon)' type_id=226
> > ...
> > [5189] PTR '(anon)' type_id=5183
> > ...
> > [5321] FUNC_PROTO '(anon)' ret_type_id=42 vlen=1
> > '__unused' type_id=5189
> > ...
> > [17648] FUNC '__x64_sys_recvmsg' type_id=5321 linkage=static
> > ...
> > ```
> >
> > Yet another distribution/kernel/syscall combination has multiple `FUNC
> > '__x64_sys_[SYSCALL]'` entries, one for FWD and the other for STRUCT:
> >
> > ```
> > $ uname -a
> > Linux ip-10-5-0-115.ap-northeast-1.compute.internal
> > 5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022
> > x86_64 x86_64 x86_64 GNU/Linux
> >
> > ```
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep
> > __x64_sys_mprotect
> > ...
> > [175] STRUCT 'pt_regs' size=168 vlen=21
> > 'r15' type_id=2 bits_offset=0
> > 'r14' type_id=2 bits_offset=64
> > 'r13' type_id=2 bits_offset=128
> > 'r12' type_id=2 bits_offset=192
> > 'bp' type_id=2 bits_offset=256
> > ...
> > [4215] CONST '(anon)' type_id=175
> > ...
> > [4220] PTR '(anon)' type_id=4215
> > ...
> > [6062] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> > 'regs' type_id=4220
> > ...
> > [11461] FWD 'pt_regs' fwd_kind=struct
> > [11462] CONST '(anon)' type_id=11461
> > [11463] PTR '(anon)' type_id=11462
> > [11464] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> > '__unused' type_id=11463
> > ...
> > [11698] FUNC '__x64_sys_mprotect' type_id=11464 linkage=static
> > ...
> > [23528] FUNC '__x64_sys_mprotect' type_id=6062 linkage=static
> > ...
> > ```
> >
> > Trying to read `regs` parameter with FWD entry results in "invalid
> > bpf_context access" error:
> >
> > ```
> > SEC("fentry/__x64_sys_recvfrom")
> > int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> > struct event t;
> >
> > bpf_get_current_comm(t.comm, TASK_COMM_LEN);
> >
> > u64 id = bpf_get_current_pid_tgid();
> > t.pid = id >> 32;
> >
> > // This causes an error on some environments.
> > t.fd = PT_REGS_PARM1_CORE(regs);
> >
> > bpf_printk("comm: %s, pid: %d, fd: %d", t.comm, t.pid, t.fd);
> >
> > return 0;
> > ```
> >
> > ```
> > $ sudo ./output
> > 2022/07/01 03:33:01 loading objects: field FentrySyscall: program
> > fentry_syscall: load program: permission denied:
> > arg#0 type is not a struct
> > Unrecognized arg#0 type PTR
> > ; int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> > 0: (79) r6 = *(u64 *)(r1 +0)
> > func '__x64_sys_recvfrom' arg0 type FWD is not a struct
> > invalid bpf_context access off=0 size=8
> > processed 1 insns (limit 1000000) max_states_per_insn 0
> > total_states 0 peak_states 0 mark_read 0
> > ```
> >
> > Is this a bug related to toolchain?
>
> nice, I think it's specific to each object that defines syscall
>
> if such object has 'struct pt_regs' header with definition included
> it will have full struct pt_regs, if not it will be just fwd ref
>
> not sure this would break anything else, but change below
> fixes it for me
hi,
did this help? I'll send it as formal patch anyway,
since it fixes the issue for me
jirka
>
> jirka
>
>
> ---
> diff --git a/arch/x86/include/asm/syscall_wrapper.h b/arch/x86/include/asm/syscall_wrapper.h
> index 59358d1bf880..fd2669b1cb2d 100644
> --- a/arch/x86/include/asm/syscall_wrapper.h
> +++ b/arch/x86/include/asm/syscall_wrapper.h
> @@ -6,7 +6,7 @@
> #ifndef _ASM_X86_SYSCALL_WRAPPER_H
> #define _ASM_X86_SYSCALL_WRAPPER_H
>
> -struct pt_regs;
> +#include <asm/ptrace.h>
>
> extern long __x64_sys_ni_syscall(const struct pt_regs *regs);
> extern long __ia32_sys_ni_syscall(const struct pt_regs *regs);
