Re: [QUESTION] bpf/tc verifier error: invalid access to map value, min value is outside of the allowed memory range

[Justin Iurman <justin.iurman@xxxxxxxxx>]

On 8/26/23 02:49, John Fastabend wrote:
> Justin Iurman wrote:
> > Hello,
> >
> > I'm facing a verifier error and don't know how to make it happy (already
> > tried lots of checks). First, here is my env:
> >    - OS: Ubuntu 22.04.3 LTS
> >    - kernel: 5.15.0-79-generic x86_64 (CONFIG_DEBUG_INFO_BTF=y)
> >    - clang version: 14.0.0-1ubuntu1.1
> >    - iproute2-5.15.0 with libbpf 0.5.0
> >
> > And here is a simplified example of my program (basically, it will
> > insert in packets some bytes defined inside a map):
> >
> > #include "vmlinux.h"
> > #include <bpf/bpf_endian.h>
> > #include <bpf/bpf_helpers.h>
> >
> > #define MAX_BYTES 2048
> >
> > struct xxx_t {
> > 	__u32 bytes_len;
> > 	__u8 bytes[MAX_BYTES];
> > };
> >
> > struct {
> > 	__uint(type, BPF_MAP_TYPE_ARRAY);
> > 	__uint(max_entries, 1);
> > 	__type(key, __u32);
> > 	__type(value, struct xxx_t);
> > 	__uint(pinning, LIBBPF_PIN_BY_NAME);
> > } my_map SEC(".maps");
> >
> > char _license[] SEC("license") = "GPL";
> >
> > SEC("egress")
> > int egress_handler(struct __sk_buff *skb)
> > {
> > 	void *data_end = (void *)(long)skb->data_end;
> > 	void *data = (void *)(long)skb->data;
> > 	struct ethhdr *eth = data;
> > 	struct ipv6hdr *ip6;
> > 	struct xxx_t *x;
> > 	__u32 offset;
> > 	__u32 idx = 0;
> >
> > 	offset = sizeof(*eth) + sizeof(*ip6);
> > 	if (data + offset > data_end)
> > 		return TC_ACT_OK;
> >
> > 	if (bpf_ntohs(eth->h_proto) != ETH_P_IPV6)
> > 		return TC_ACT_OK;
> >
> > 	x = bpf_map_lookup_elem(&my_map, &idx);
> > 	if (!x)
> > 		return TC_ACT_OK;
> >
> > 	if (x->bytes_len == 0 || x->bytes_len > MAX_BYTES)
> > 		return TC_ACT_OK;

FYI, with bytes_len on the stack (no direct access) and a min check 
above 1, it works (I thought I already tried it, but I suspect I did on 
direct access only). What's funny though is that the verifier still 
fails on the exact same code with a recent kernel.

So, "bytes_len == 0" or "bytes_len < 1" doesn't work, but "bytes_len < 
2" works (which is weird). Fortunately, I need a min value of 8, which 
makes the hack OK, but I'm still trying to understand what's going on 
here. And, as mentioned, it does not work with direct access (i.e., 
"x->bytes_len < 2").

Thanks,
Justin

> > 	if (bpf_skb_adjust_room(skb, x->bytes_len, BPF_ADJ_ROOM_NET, 0))
> > 		return TC_ACT_OK;
> >
> > 	if (bpf_skb_store_bytes(skb, offset, x->bytes, 8/*x->bytes_len*/,
>
> You will see lots of folks & that value with something to
> ensure compiler/verifier get a solid upper/lower bounds.
> This is slightly kernel dependent the newer kernels are
> better at tracking bounds.
>
> This should do what you want more or less,
>
>    x->bytes_len &= 0x7ff