Hey Eduard!
On Wed, Feb 22, 2023 at 05:28:52PM +0200, Eduard Zingerman wrote:
> On Mon, 2023-02-20 at 22:35 +0000, Matt Bobrowski wrote:
> > Hello!
> >
> > Whilst in the midst of testing a v5.19 to v6.1 kernel upgrade, we
> > happened to notice that one of our sleepable LSM based eBPF programs
> > was failing to load on the newer v6.1 kernel. Using the below trivial
> > eBPF program as our reproducer:
> >
> > #include "vmlinux.h"
> > #include <bpf/bpf_helpers.h>
> > #include <bpf/bpf_tracing.h>
> >
> > char LICENSE[] SEC("license") = "Dual BSD/GPL";
> >
> > SEC("lsm.s/bprm_committed_creds")
> > int BPF_PROG(dbg, struct linux_binprm *bprm)
> > {
> > char buf[64] = {0};
> > bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
> > return 0;
> > }
> >
> > The verifier emits the following error message when attempting to load
> > the above eBPF program:
> >
> > -- BEGIN PROG LOAD LOG --
> > reg type unsupported for arg#0 function dbg#5
> > 0: R1=ctx(off=0,imm=0) R10=fp0
> > ; int BPF_PROG(dbg, struct linux_binprm *bprm)
> > 0: (79) r1 = *(u64 *)(r1 +0)
> > func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 137293 type STRUCT 'linux_binprm'
> > 1: R1_w=ptr_linux_binprm(off=0,imm=0)
> > 1: (b7) r2 = 0 ; R2_w=0
> > ; char buf[64] = {0};
> > [...]
> > ; bpf_ima_file_hash(bprm->file, buf, 64);
> > 10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
> > 11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
> > ;
> > 12: (07) r2 += -64 ; R2_w=fp-64
> > ; bpf_ima_file_hash(bprm->file, buf, 64);
> > 13: (b7) r3 = 64 ; R3_w=64
> > 14: (85) call bpf_ima_file_hash#193
> > cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1
> > R1 is of type file but file is expected
> > processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
> > -- END PROG LOAD LOG --
> >
> > What particularly strikes out at me is the following 2 lines returned
> > in the error message:
> >
> > cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1
> > R1 is of type file but file is expected
>
> Hi Matt,
>
> I tried your program as a ./test_progs test using v6.1 kernel and
> don't see any error messages:
>
> VERIFIER LOG:
> =============
> func#0 @0
> reg type unsupported for arg#0 function dbg#5
> 0: R1=ctx(off=0,imm=0) R10=fp0
> ; int BPF_PROG(dbg, struct linux_binprm *bprm)
> 0: (79) r1 = *(u64 *)(r1 +0)
> func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 3061 type STRUCT 'linux_binprm'
> 1: R1_w=ptr_linux_binprm(off=0,imm=0)
> 1: (b7) r2 = 0 ; R2_w=0
> ; char buf[64] = {0};
> 2: (7b) *(u64 *)(r10 -8) = r2
> last_idx 2 first_idx 0
> regs=4 stack=0 before 1: (b7) r2 = 0
> 3: R2_w=P0 R10=fp0 fp-8_w=00000000
> 3: (7b) *(u64 *)(r10 -16) = r2 ; R2_w=P0 R10=fp0 fp-16_w=00000000
> 4: (7b) *(u64 *)(r10 -24) = r2 ; R2_w=P0 R10=fp0 fp-24_w=00000000
> 5: (7b) *(u64 *)(r10 -32) = r2 ; R2_w=P0 R10=fp0 fp-32_w=00000000
> 6: (7b) *(u64 *)(r10 -40) = r2 ; R2_w=P0 R10=fp0 fp-40_w=00000000
> 7: (7b) *(u64 *)(r10 -48) = r2 ; R2_w=P0 R10=fp0 fp-48_w=00000000
> 8: (7b) *(u64 *)(r10 -56) = r2 ; R2_w=P0 R10=fp0 fp-56_w=00000000
> 9: (7b) *(u64 *)(r10 -64) = r2 ; R2_w=P0 R10=fp0 fp-64_w=00000000
> ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
> 10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
> 11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
> ;
> 12: (07) r2 += -64 ; R2_w=fp-64
> ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
> 13: (b4) w3 = 64 ; R3_w=64
> 14: (85) call bpf_ima_file_hash#193
> last_idx 14 first_idx 0
> regs=8 stack=0 before 13: (b4) w3 = 64
> 15: R0_w=scalar() fp-8_w=mmmmmmmm fp-16_w=mmmmmmmm fp-24_w=mmmmmmmm fp-32_w=mmmmmmmm fp-40_w=mmmmmmmm fp-48_w=mmmmmmmm fp-56_w=mmmmmmmm fp-64_w=mmmmmmmm
> ; int BPF_PROG(dbg, struct linux_binprm *bprm)
> 15: (b4) w0 = 0 ; R0_w=0
> 16: (95) exit
>
> I use the following revision: 830b3c68c1fb "Linux 6.1".
> (also works with current bpf-next master).
>
> Could you please provide some details on how you compile/load the program?
Firstly, thanks for taking a peek at this! Secondly, I do apologies, I
should've provided some more detailed on how I'm reproducing this in
my initial email. Below you can find a transcript of how I'm
conducting my tests:
The source OS which things (kernel and BPF reproducer program) are
being built on:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux rodete"
NAME="Debian GNU/Linux"
VERSION_ID="rodete"
VERSION="12 (rodete)"
VERSION_CODENAME=rodete
ID=debian
Building latest LLVM and Pahole from source:
$ sudo apt install -y cmake
$ cmake --version
cmake version 3.25.1
$ git clone https://github.com/llvm/llvm-project.git && cd llvm-project && \
mkdir build && \
cd build && \
cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE=Release -G "Unix Makefiles" ../llvm && \
make -j $(nproc) && \
sudo make install
$ clang --version
clang version 17.0.0 (https://github.com/llvm/llvm-project.git bc85cf1687435f28fb01b1aa5303317e6118490c)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/bin
$ sudo apt install -y libdwarf-dev libdw-dev
$ git clone git://git.kernel.org/pub/scm/devel/pahole/pahole.git && \
cd pahole && \
mkdir build && \
cd build && \
cmake -DCMAKE_INSTALL_PREFIX=/usr -D__LIB=lib .. && \
make -j $(nproc) && \
sudo make install
$ pahole --version
v1.25
Building a test kernel:
$ git clone https://github.com/torvalds/linux.git && cd linux
$ make defconfig && make kvm_guest.config
$ scripts/config \
-e BPF \
-e BPF_SYSCALL \
-e BPF_LSM \
-e BPF_JIT \
-e BPF_EVENTS \
-e DEBUG_INFO \
-e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
-e DEBUG_INFO_BTF \
-e DEBUG_INFO_BTF_MODULES \
-e PAHOLE_HAS_SPLIT_BTF \
-e FTRACE \
-e DYNAMIC_FTRACE \
-e FUNCTION_TRACER
$ make olddefconfig
$ make -j`nproc`
Building the BPF reproducer program:
$ git clone https://github.com/libbpf/libbpf-bootstrap.git
# Both libbpf and bpftool should be the latest versions here.
$ git submodule update --init --recursive
$ cd examples/c
$ cat > fentry.bpf.c<<EOF
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char LICENSE[] SEC("license") = "Dual BSD/GPL";
SEC("lsm.s/bprm_committed_creds")
int BPF_PROG(dbg, struct linux_binprm *bprm)
{
char buf[64] = {0};
bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
return 0;
}
EOF
$ make -j`nproc` fentry
At this point, I basically launch the built kernel using QEMU and push
the built 'fentry' BPF program to the VM and run it. At that point, I
face the BPF verifier issue.
LMK whether you need any more information.
/M
