On Wed, Aug 16, 2023 at 9:08 AM Arnaldo Carvalho de Melo
<acme@xxxxxxxxxx> wrote:
>
> Em Wed, Aug 16, 2023 at 10:22:15AM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Wed, Aug 16, 2023 at 10:11:11AM -0300, Arnaldo Carvalho de Melo escreveu:
> > > Just taking notes about things to work on top of what is in
> > > tmp.perf-tools-next, that will move to perf-tools-next soon:
>
> > > We need to make these libbpf error messages appear only in verbose mode,
> > > and probably have a hint about unprivileged BPF, a quick attempt failed
> > > after several attempts at getting privileges :-\
>
> > > Probably attaching to tracepoints is off limits to !root even with
> > > /proc/sys/kernel/unprivileged_bpf_disabled set to zero.
>
> > yep, the libbpf sys_bpf call to check if it could load a basic BPF
> > bytecode (prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2) succeeds,
> > but then, later we manage to create the maps, etc to then stumble on
>
> > bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=8272, max_entries=1, map_flags=0, inner_map_fd=0, map_name="augmented_args_", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 72) = 7
> > bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\t\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=81, btf_log_size=0, btf_log_level=0}, 32) = -1 EPERM (Operation not permitted)
>
> > and:
>
> > bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=2, insns=0x1758340, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(6, 4, 7), prog_flags=0, prog_name="syscall_unaugme", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = -1 EPERM (Operation not permitted)
>
> > So 'perf trace' should just not try to load the augmented_raw_syscalls
> > BPF skel for !root.
>
> Not really, I insisted and it is (was?) possible to make it work,
> testing on some other machine and after having to change the permissions
> recursively on tracefs (before a remount with mode=755 seemed to
> work?).
>
> I managed to make it work for !root, BPF collecting the pointer args for
> openat, access (perf trace looks for syscall signatures and reuses BPF
> progs for the ones matching one of the explicitely provided)
> clock_namosleep, etc.
>
> (re)Reading Documentation/admin-guide/perf-security.rst and getting it
> into the hints system of 'perf trace' may make this process simpler and
> safer, by using a group, etc. But it is possible, great!
>
> I didn't even had to touch /proc/sys/kernel/unprivileged_bpf_disabled,
> just the capabilities for the perf binary (which is a pretty big window,
> but way smaller than touching /proc/sys/kernel/unprivileged_bpf_disabled).
>
> So now we need to get BUILD_BPF_SKEL=1 to be the default but just emit a
> warning when what is needed isn't available, just like with other
> features, in that case 'perf trace' continues as today, no pointer arg
> contents collection.
>
> Unfortunately it is too late in the process for v6.6 even, so as soon as
> perf-tools-next becomes perf-tools and we reopen it for v6.7 the first
> patch should be this build BPF skels if what is needed is available.
Thanks Arnaldo! I think targeting this for 6.7 makes sense and thanks
for diving around all the permission issues.
Ian
> I'll also check if we can enable BUILD_BPF_SKEL=1 in the distro packages
> so that we collect some info from them about possible problems.
>
> What I have is now in perf-tools-next, so should get into linux-next and
> hopefully help in testing it, IIRC there are CIs that enable
> BUILD_BPF_SKEL=1.
>
> - Arnaldo
>
> [acme@five ~]$ uname -a
> Linux five 6.2.15-100.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 11 16:51:53 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
> [acme@five ~]$ id
> uid=1000(acme) gid=1000(acme) groups=1000(acme),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> [acme@five ~]$ perf trace sleep 1
> ? ( ): sleep/980735 ... [continued]: execve()) = 0
> 0.031 ( 0.002 ms): sleep/980735 brk() = 0x55c621548000
> 0.039 ( 0.001 ms): sleep/980735 arch_prctl(option: 0x3001, arg2: 0x7ffeb8a6a460) = -1 EINVAL (Invalid argument)
> 0.058 ( 0.006 ms): sleep/980735 access(filename: "/etc/ld.so.preload", mode: R) = -1 ENOENT (No such file or directory)
> 0.068 ( 0.005 ms): sleep/980735 openat(dfd: CWD, filename: "/etc/ld.so.cache", flags: RDONLY|CLOEXEC) = 3
> 0.074 ( 0.002 ms): sleep/980735 newfstatat(dfd: 3, filename: "", statbuf: 0x7ffeb8a69680, flag: 4096) = 0
> 0.077 ( 0.006 ms): sleep/980735 mmap(len: 54771, prot: READ, flags: PRIVATE, fd: 3) = 0x7f6b95ad9000
> 0.084 ( 0.001 ms): sleep/980735 close(fd: 3) = 0
> 0.094 ( 0.006 ms): sleep/980735 openat(dfd: CWD, filename: "/lib64/libc.so.6", flags: RDONLY|CLOEXEC) = 3
> 0.101 ( 0.002 ms): sleep/980735 read(fd: 3, buf: 0x7ffeb8a697e8, count: 832) = 832
> 0.105 ( 0.001 ms): sleep/980735 pread64(fd: 3, buf: 0x7ffeb8a693e0, count: 784, pos: 64) = 784
> 0.107 ( 0.001 ms): sleep/980735 pread64(fd: 3, buf: 0x7ffeb8a69380, count: 80, pos: 848) = 80
> 0.110 ( 0.001 ms): sleep/980735 pread64(fd: 3, buf: 0x7ffeb8a69330, count: 68, pos: 928) = 68
> 0.113 ( 0.002 ms): sleep/980735 newfstatat(dfd: 3, filename: "", statbuf: 0x7ffeb8a69680, flag: 4096) = 0
> 0.115 ( 0.003 ms): sleep/980735 mmap(len: 8192, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS) = 0x7f6b95ad7000
> 0.122 ( 0.001 ms): sleep/980735 pread64(fd: 3, buf: 0x7ffeb8a692d0, count: 784, pos: 64) = 784
> 0.126 ( 0.006 ms): sleep/980735 mmap(len: 2104720, prot: READ, flags: PRIVATE|DENYWRITE, fd: 3) = 0x7f6b95800000
> 0.133 ( 0.013 ms): sleep/980735 mmap(addr: 0x7f6b95828000, len: 1523712, prot: READ|EXEC, flags: PRIVATE|FIXED|DENYWRITE, fd: 3, off: 0x28000) = 0x7f6b95828000
> 0.147 ( 0.008 ms): sleep/980735 mmap(addr: 0x7f6b9599c000, len: 360448, prot: READ, flags: PRIVATE|FIXED|DENYWRITE, fd: 3, off: 0x19c000) = 0x7f6b9599c000
> 0.156 ( 0.010 ms): sleep/980735 mmap(addr: 0x7f6b959f4000, len: 24576, prot: READ|WRITE, flags: PRIVATE|FIXED|DENYWRITE, fd: 3, off: 0x1f3000) = 0x7f6b959f4000
> 0.171 ( 0.005 ms): sleep/980735 mmap(addr: 0x7f6b959fa000, len: 32144, prot: READ|WRITE, flags: PRIVATE|FIXED|ANONYMOUS) = 0x7f6b959fa000
> 0.182 ( 0.001 ms): sleep/980735 close(fd: 3) = 0
> 0.193 ( 0.003 ms): sleep/980735 mmap(len: 12288, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS) = 0x7f6b95ad4000
> 0.199 ( 0.001 ms): sleep/980735 arch_prctl(option: SET_FS, arg2: 0x7f6b95ad4740) = 0
> 0.202 ( 0.001 ms): sleep/980735 set_tid_address(tidptr: 0x7f6b95ad4a10) = 980735 (sleep)
> 0.204 ( 0.001 ms): sleep/980735 set_robust_list(head: 0x7f6b95ad4a20, len: 24) = 0
> 0.206 ( 0.001 ms): sleep/980735 rseq(rseq: 0x7f6b95ad50e0, rseq_len: 32, sig: 1392848979) = 0
> 0.277 ( 0.010 ms): sleep/980735 mprotect(start: 0x7f6b959f4000, len: 16384, prot: READ) = 0
> 0.306 ( 0.007 ms): sleep/980735 mprotect(start: 0x55c61fa4a000, len: 4096, prot: READ) = 0
> 0.320 ( 0.010 ms): sleep/980735 mprotect(start: 0x7f6b95b1c000, len: 8192, prot: READ) = 0
> 0.340 ( 0.002 ms): sleep/980735 prlimit64(resource: STACK, old_rlim: 0x7ffeb8a6a1c0) = 0
> 0.349 ( 0.009 ms): sleep/980735 munmap(addr: 0x7f6b95ad9000, len: 54771) = 0
> 0.381 ( 0.002 ms): sleep/980735 getrandom(ubuf: 0x7f6b959ff4d8, len: 8, flags: NONBLOCK) = 8
> 0.386 ( 0.001 ms): sleep/980735 brk() = 0x55c621548000
> 0.388 ( 0.006 ms): sleep/980735 brk(brk: 0x55c621569000) = 0x55c621569000
> 0.403 ( 0.012 ms): sleep/980735 openat(dfd: CWD, filename: "", flags: RDONLY|CLOEXEC) = 3
> 0.417 ( 0.003 ms): sleep/980735 newfstatat(dfd: 3, filename: "", statbuf: 0x7f6b959f9b80, flag: 4096) = 0
> 0.422 ( 0.008 ms): sleep/980735 mmap(len: 224096080, prot: READ, flags: PRIVATE, fd: 3) = 0x7f6b88200000
> 0.436 ( 0.002 ms): sleep/980735 close(fd: 3) = 0
> 0.480 (1000.041 ms): sleep/980735 clock_nanosleep(rqtp: { .tv_sec: 1, .tv_nsec: 0 }, rmtp: 0x7ffeb8a6a450) = 0
> 1000.552 ( 0.003 ms): sleep/980735 close(fd: 1) = 0
> 1000.558 ( 0.002 ms): sleep/980735 close(fd: 2) = 0
> 1000.565 ( ): sleep/980735 exit_group() = ?
> [acme@five ~]$ getcap ~/bin/perf
> /var/home/acme/bin/perf cap_perfmon,cap_bpf=ep
> [acme@five ~]$ cat /proc/sys/kernel/unprivileged_bpf_disabled
> 2
> [acme@five ~]$ cat /proc/sys/kernel/perf_event_paranoid
> -1
> [acme@five ~]$
