We’ve been introducing bpf_tail_call’s into our XDP programs and have run into
packet loss and latency increases when performing load tests. After profiling
our code we’ve come to the conclusion that this is the problem area in our code:
`int layer3_protocol = bpf_ntohs(ethernet_header->h_proto);`
This is the first time we read from the packet in the first XDP program. We have
yet to make a tail call at this point. However, we do write into the metadata
section prior to this line.
How We Profiled Our Code:
To profile our code, we used https://github.com/iovisor/bpftrace. We ran this
command while sending traffic to our machine:
`sudo bpftrace bpftrace -e 'profile:hz:99 { @[kstack] = count(); }' >
/tmp/stack_samples.out`
>From there we got a kernel stack trace with the most frequently counted spots at
the bottom of the output file. The most commonly hit spot asides from the CPU
idle look like:
```
@[
bpf_prog_986b0b3beb6f0873_some_program+290
i40e_napi_poll+1897
net_rx_action+309
__softirqentry_text_start+202
run_ksoftirqd+38
smpboot_thread_fn+197
kthread+283
ret_from_fork+34
]: 8748
```
We then took the program id and ran this command to retrieve the jited code:
`sudo bpftool prog dump jited tag 986b0b3beb6f0873`
By converting the decimal offset (290) from the stack trace to hex format (122)
we found the line which it’s referring to in the jited code:
```
11d: movzbq 0xc(%r15),%rsi
122: movzbq 0xd(%r15),%rdi
127: shl $0x8,%rdi
12b: or %rsi,%rdi
12e: ror $0x8,%di
132: movzwl %di,%edi
```
We've mapped this portion to refer to the line mentioned earlier:
`int layer3_protocol = bpf_ntohs(ethernet_header->h_proto);`
1) Are we correctly profiling our XDP programs?
2) Is there a reason why our first read into the packet would cause this issue?
And what would be the best way to solve the issue?
We've theorized it may have to do with cache or TLB misses as we've added a lot
more instructions to our programs.
Thanks for your time!
|