Hello,
CC'ing netdev as well, since I initially suspected an issue in iproute2.
However, after having recompiled iproute2 with libbpf, I'm still stuck
facing the same problem.
Environment:
- OS: Ubuntu 20.04.5 LTS
- kernel: 5.4.0-137-generic x86_64 (CONFIG_DEBUG_INFO_BTF=y)
- clang version 10.0.0-4ubuntu1
- iproute2-6.1.0, libbpf 1.2.0 (iproute2-ss200127 installed by default
without libbpf)
Note: same result with kernel 6.2.0-rc6+ (net-next), as a test to be
aligned with latest iproute2 version (just in case).
Long story short: I can't for the life of me make the ebpf program load
correctly with tc. What's the cause? Well, a map, and the verifier
doesn't like it. I must definitely be doing something wrong, but can't
find what. Here is a reproducible and minimal example:
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#define TC_ACT_OK 0
#define MAX_BYTES 2048
char _license[] SEC("license") = "GPL";
struct mystruct_t {
__u8 bytes[MAX_BYTES];
};
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(key_size, sizeof(__u8));
__uint(value_size, sizeof(struct mystruct_t));
__uint(max_entries, 1);
} percpu_map SEC(".maps");
SEC("egress")
int xxx(struct __sk_buff *skb)
{
__u8 idx = 0;
struct mystruct_t *x = bpf_map_lookup_elem(&percpu_map, &idx);
return TC_ACT_OK;
}
Here is how I compile the whole thing:
git clone --recursive --depth 1 https://github.com/libbpf/libbpf
./deps/libbpf
make -j -C deps/libbpf/src/ BUILD_STATIC_ONLY=y DESTDIR="build"
INCLUDEDIR= LIBDIR= UAPIDIR= install
git clone --recursive --depth 1 https://github.com/libbpf/bpftool
./deps/bpftool
make -j -C deps/bpftool/src/
deps/bpftool/src/bpftool btf dump file /sys/kernel/btf/vmlinux format c
> build/vmlinux.h
clang -g -O2 -Wall -Wextra -target bpf -D__TARGET_ARCH_x86_64 -I build/
-c program.c -o build/program.o
I noticed that "clang-bpf-co-re" is OFF when compiling bpftool, don't
know if it's part of the problem or not. Here is what the "build"
directory looks like after that:
$ ls -al build
[...]
drwxr-xr-x 2 justin justin 4096 fév 6 18:20 bpf
-rw-rw-r-- 1 justin justin 3936504 fév 6 18:20 libbpf.a
drwxr-xr-x 2 justin justin 4096 fév 6 18:20 pkgconfig
-rw-rw-r-- 1 justin justin 10592 fév 7 10:42 program.o
-rw-rw-r-- 1 justin justin 2467774 fév 7 10:42 vmlinux.h
And here is the verifier error I got when loading it with tc (qdisc
clsact already attached):
$ sudo ../deps/iproute2/tc/tc filter add dev eno2 egress bpf da obj
program.o sec egress
libbpf: map 'percpu_map': failed to create: Invalid argument(-22)
libbpf: failed to load object 'program.o'
Unable to load program
If I compile without the -g, of course it complains about
missing/corrupted BTF. I received a little bit more info with "legacy"
(= w/o libbpf) iproute2 (iproute2-ss200127):
$ sudo tc filter add dev eno2 egress bpf da obj program.o sec egress
BTF debug data section '.BTF' rejected: Invalid argument (22)!
- Length: 1628
Verifier analysis:
magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 948
str_off: 948
str_len: 656
btf_total_size: 1628
[1] STRUCT (anon) size=32 vlen=4
type type_id=2 bits_offset=0
key_size type_id=6 bits_offset=64
value_size type_id=8 bits_offset=128
max_entries type_id=6 bits_offset=192
[2] PTR (anon) type_id=4
[3] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[4] ARRAY (anon) type_id=3 index_type_id=5 nr_elems=6
[5] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)
[6] PTR (anon) type_id=7
[7] ARRAY (anon) type_id=3 index_type_id=5 nr_elems=1
[8] PTR (anon) type_id=9
[9] ARRAY (anon) type_id=3 index_type_id=5 nr_elems=2048
[10] VAR percpu_map type_id=1 linkage=1
[11] PTR (anon) type_id=12
[12] STRUCT __sk_buff size=176 vlen=31
len type_id=13 bits_offset=0
pkt_type type_id=13 bits_offset=32
mark type_id=13 bits_offset=64
queue_mapping type_id=13 bits_offset=96
protocol type_id=13 bits_offset=128
vlan_present type_id=13 bits_offset=160
vlan_tci type_id=13 bits_offset=192
vlan_proto type_id=13 bits_offset=224
priority type_id=13 bits_offset=256
ingress_ifindex type_id=13 bits_offset=288
ifindex type_id=13 bits_offset=320
tc_index type_id=13 bits_offset=352
cb type_id=15 bits_offset=384
hash type_id=13 bits_offset=544
tc_classid type_id=13 bits_offset=576
data type_id=13 bits_offset=608
data_end type_id=13 bits_offset=640
napi_id type_id=13 bits_offset=672
family type_id=13 bits_offset=704
remote_ip4 type_id=13 bits_offset=736
local_ip4 type_id=13 bits_offset=768
remote_ip6 type_id=16 bits_offset=800
local_ip6 type_id=16 bits_offset=928
remote_port type_id=13 bits_offset=1056
local_port type_id=13 bits_offset=1088
data_meta type_id=13 bits_offset=1120
(anon) type_id=17 bits_offset=1152
tstamp type_id=19 bits_offset=1216
wire_len type_id=13 bits_offset=1280
gso_segs type_id=13 bits_offset=1312
(anon) type_id=21 bits_offset=1344
[13] TYPEDEF __u32 type_id=14
[14] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[15] ARRAY (anon) type_id=13 index_type_id=5 nr_elems=5
[16] ARRAY (anon) type_id=13 index_type_id=5 nr_elems=4
[17] UNION (anon) size=8 vlen=1
flow_keys type_id=18 bits_offset=0
[18] PTR (anon) type_id=30
[19] TYPEDEF __u64 type_id=20
[20] INT long long unsigned int size=8 bits_offset=0 nr_bits=64
encoding=(none)
[21] UNION (anon) size=8 vlen=1
sk type_id=22 bits_offset=0
[22] PTR (anon) type_id=31
[23] FUNC_PROTO (anon) return=3 args=(11 skb)
[24] FUNC xxx type_id=23 vlen != 0
Prog section 'egress' rejected: Permission denied (13)!
- Type: 3
- Instructions: 13 (0 over limit)
- License: GPL
Verifier analysis:
0: (b7) r1 = 0
1: (73) *(u8 *)(r10 -1) = r1
last_idx 1 first_idx 0
regs=2 stack=0 before 0: (b7) r1 = 0
2: (bf) r2 = r10
3: (07) r2 += -1
4: (18) r1 = 0x0
6: (85) call bpf_map_lookup_elem#1
R1 type=inv expected=map_ptr
processed 6 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0
Error fetching program/map!
Unable to load program
So here, basically, the call to bpf_map_lookup_elem fails because the
first argument is not a map_ptr (obviously r1 = 0x0), which is why I
suspected some difference with iproute2. However, it looks like it is
the same with iproute2 compiled with libbpf (is it?). But I don't know
if it's reliable because that error appears when using tc w/o libbpf. As
a FYI, I had the exact same issue with a global variable that I do not
need anymore, by the way.
My question is the following: what am I doing wrong? I'm not that used
to ebpf programming, so there must be something I'm missing.
