Re: [QUESTION] bpf/tc verifier error: invalid access to map value, min value is outside of the allowed memory range

[Justin Iurman <justin.iurman@xxxxxxxxx>]

On 8/26/23 02:49, John Fastabend wrote:
> Justin Iurman wrote:
> > Hello,
> >
> > I'm facing a verifier error and don't know how to make it happy (already
> > tried lots of checks). First, here is my env:
> >    - OS: Ubuntu 22.04.3 LTS
> >    - kernel: 5.15.0-79-generic x86_64 (CONFIG_DEBUG_INFO_BTF=y)
> >    - clang version: 14.0.0-1ubuntu1.1
> >    - iproute2-5.15.0 with libbpf 0.5.0
> >
> > And here is a simplified example of my program (basically, it will
> > insert in packets some bytes defined inside a map):
> >
> > #include "vmlinux.h"
> > #include <bpf/bpf_endian.h>
> > #include <bpf/bpf_helpers.h>
> >
> > #define MAX_BYTES 2048
> >
> > struct xxx_t {
> > 	__u32 bytes_len;
> > 	__u8 bytes[MAX_BYTES];
> > };
> >
> > struct {
> > 	__uint(type, BPF_MAP_TYPE_ARRAY);
> > 	__uint(max_entries, 1);
> > 	__type(key, __u32);
> > 	__type(value, struct xxx_t);
> > 	__uint(pinning, LIBBPF_PIN_BY_NAME);
> > } my_map SEC(".maps");
> >
> > char _license[] SEC("license") = "GPL";
> >
> > SEC("egress")
> > int egress_handler(struct __sk_buff *skb)
> > {
> > 	void *data_end = (void *)(long)skb->data_end;
> > 	void *data = (void *)(long)skb->data;
> > 	struct ethhdr *eth = data;
> > 	struct ipv6hdr *ip6;
> > 	struct xxx_t *x;
> > 	__u32 offset;
> > 	__u32 idx = 0;
> >
> > 	offset = sizeof(*eth) + sizeof(*ip6);
> > 	if (data + offset > data_end)
> > 		return TC_ACT_OK;
> >
> > 	if (bpf_ntohs(eth->h_proto) != ETH_P_IPV6)
> > 		return TC_ACT_OK;
> >
> > 	x = bpf_map_lookup_elem(&my_map, &idx);
> > 	if (!x)
> > 		return TC_ACT_OK;
> >
> > 	if (x->bytes_len == 0 || x->bytes_len > MAX_BYTES)
> > 		return TC_ACT_OK;
> >
> > 	if (bpf_skb_adjust_room(skb, x->bytes_len, BPF_ADJ_ROOM_NET, 0))
> > 		return TC_ACT_OK;
> >
> > 	if (bpf_skb_store_bytes(skb, offset, x->bytes, 8/*x->bytes_len*/,
>
> You will see lots of folks & that value with something to
> ensure compiler/verifier get a solid upper/lower bounds.
> This is slightly kernel dependent the newer kernels are
> better at tracking bounds.
>
> This should do what you want more or less,
>
>    x->bytes_len &= 0x7ff

John,

Thanks for this. My bad, I should have explicitly listed the checks I 
already tried. In fact, what you mentioned above, as well as having the 
length on the stack, are part of my unsuccessful attempts. For instance, 
having the length on the stack combined with the min/max check gives the 
same result as x->bytes_len &= 0xfff:

R4_w=inv(id=2,umax_value=2048,var_off=(0x0; 0xfff))

... which is looking good (but still, no min value, which seems to be 
the issue here). More of it, when the length is on the stack:

; if (bpf_skb_store_bytes(skb, offset, x->bytes, len,
29: (bf) r1 = r7
30: (b7) r2 = 54
31: (bf) r3 = r8
32: (bf) r4 = r9
33: (b7) r5 = 1
34: (85) call bpf_skb_store_bytes#9
 R0=inv0 R1_w=ctx(id=0,off=0,imm=0) R2_w=inv54 
R3_w=map_value(id=0,off=4,ks=4,vs=2052,imm=0) 
R4_w=inv(id=2,umax_value=2048,var_off=(0x0; 0xfff)) R5_w=inv1 R6_w=inv1 
R7=ctx(id=0,off=0,imm=0) R8_w=map_value(id=0,off=4,ks=4,vs=2052,imm=0) 
R9=inv(id=2,umax_value=2048,var_off=(0x0; 0xfff)) R10=fp0 fp-8=mmmm????
invalid access to map value, value_size=2052 off=4 size=0
R3 min value is outside of the allowed memory range

So, whatever I try, it wouldn't set the min value. Besides, it complains 
about R3 (x->bytes), while it's clearly a problem with R4 (x->bytes_len, 
or in this case "len" on the stack). As a proof, if I set x->bytes_len 
to, e.g., 8 then it works.

I'll try with a more recent kernel version and see if it makes a 
difference. Are you aware of any other hack/solution for this verifier 
error, just in case?

Thanks,
Justin