[BUG] optimizations for branch cause bpf verification to fail

[Zhongqiu Duan <dzq.aishenghu0@xxxxxxxxx>]

Hello,  everyone.

Recently, I've been doing some work using eBPF techniques. A situation was
encountered in which a program was rejected by the verifier.

Iterate over different maps under different conditions. It should be a good idea
to use map-of-maps when there are lots of maps. I use if cond for a quick test.

It looks like this:

int foo(struct xdp_md *ctx)
{
   void *data_end = (void *)(long)ctx->data_end;
   void *data = (void *)(long)ctx->data;
   struct callback_ctx cb_data;

   cb_data.output = 0;

   if (data_end - data > 1024) {
       bpf_for_each_map_elem(&map1, cb, &cb_data, 0);
   } else {
       bpf_for_each_map_elem(&map2, cb, &cb_data, 0);
   }

   if (cb_data.output)
       return XDP_DROP;

   return XDP_PASS;
}

Compile by clang-15 with optimization level O2:

0000000000000000 <foo>:
;     void *data = (void *)(long)ctx->data;
0:       61 12 00 00 00 00 00 00 r2 = *(u32 *)(r1 + 0)
;     void *data_end = (void *)(long)ctx->data_end;
1:       61 13 04 00 00 00 00 00 r3 = *(u32 *)(r1 + 4)
;     if (data_end - data > 1024) {
2:       1f 23 00 00 00 00 00 00 r3 -= r2
3:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
5:       65 03 02 00 00 04 00 00 if r3 s> 1024 goto +2 <LBB0_2>
6:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
0000000000000040 <LBB0_2>:
8:       b7 02 00 00 00 00 00 00 r2 = 0
;     cb_data.output = 0;
9:       63 2a f8 ff 00 00 00 00 *(u32 *)(r10 - 8) = r2
10:       bf a3 00 00 00 00 00 00 r3 = r10
11:       07 03 00 00 f8 ff ff ff r3 += -8
12:       18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll
14:       b7 04 00 00 00 00 00 00 r4 = 0
15:       85 00 00 00 a4 00 00 00 call 164
16:       b7 00 00 00 02 00 00 00 r0 = 2
;     if (cb_data.output)
17:       61 a1 f8 ff 00 00 00 00 r1 = *(u32 *)(r10 - 8)
; }
18:       15 01 01 00 00 00 00 00 if r1 == 0 goto +1 <LBB0_4>
19:       b7 00 00 00 01 00 00 00 r0 = 1
00000000000000a0 <LBB0_4>:
; }
20:       95 00 00 00 00 00 00 00 exit

When loading the prog, the verifier complained "tail_call abusing map_ptr".
The Compiler's optimizations look fine, so I took a quick look at the code of
the verifier.

The function record_func_map called by check_helper_call will ref the current
map in bpf_insn_aux_data of current insn. After the current branch ends,
pop stack and enter another branch, but the relevant state is not cleared.
This time, record_func_map set BPF_MAP_PTR_POISON as the map state.
At the start of set_map_elem_callback_state, poisoned state causing EINVAL.

I'm not very familiar with BPF. If it is designed like this, it is
customary to add
options on the compiler side to avoid it, then please let me know.

Thanks,
Zhongqiu