Since commit 96360004b862 ("xdp: Make devmap flush_list common for all map
instances"), devmap flushing is a global operation instead of tied to a
particular map. This means that with a bit of refactoring, we can finally fix
the performance delta between the bpf_redirect_map() and bpf_redirect() helper
functions, by introducing bulking for the latter as well.
This series makes this change by moving the data structure used for the bulking
into struct net_device itself, so we can access it even when there is not
devmap. Once this is done, moving the bpf_redirect() helper to use the bulking
mechanism becomes quite trivial, and brings bpf_redirect() up to the same as
bpf_redirect_map():
Before: After:
1 CPU:
bpf_redirect_map: 8.4 Mpps 8.4 Mpps (no change)
bpf_redirect: 5.0 Mpps 8.4 Mpps (+68%)
2 CPUs:
bpf_redirect_map: 15.9 Mpps 16.1 Mpps (+1% or ~no change)
bpf_redirect: 9.5 Mpps 15.9 Mpps (+67%)
After this patch series, the only semantics different between the two variants
of the bpf() helper (apart from the absence of a map argument, obviously) is
that the _map() variant will return an error if passed an invalid map index,
whereas the bpf_redirect() helper will succeed, but drop packets on
xdp_do_redirect(). This is because the helper has no reference to the calling
netdev, so unfortunately we can't do the ifindex lookup directly in the helper.
Changelog:
v3:
- Switch two more fields to avoid a list_head spanning two cache lines
- Include Jesper's tracepoint patch
- Also rename xdp_do_flush_map()
- Fix a few nits from Maciej
v2:
- Consolidate code paths and tracepoints for map and non-map redirect variants
(Björn)
- Add performance data for 2-CPU test (Jesper)
- Move fields to avoid shifting cache lines in struct net_device (Eric)
---
Jesper Dangaard Brouer (1):
devmap: Adjust tracepoint for map-less queue flush
Toke Høiland-Jørgensen (2):
xdp: Move devmap bulk queue into struct net_device
xdp: Use bulking for non-map XDP_REDIRECT and consolidate code paths
drivers/net/tun.c | 4 +
drivers/net/veth.c | 2 -
drivers/net/virtio_net.c | 2 -
include/linux/bpf.h | 13 +++-
include/linux/filter.h | 10 ++-
include/linux/netdevice.h | 13 ++--
include/trace/events/xdp.h | 130 +++++++++++++++++-----------------------
kernel/bpf/devmap.c | 95 +++++++++++++++--------------
net/core/dev.c | 2 +
net/core/filter.c | 90 ++++++----------------------
samples/bpf/xdp_monitor_kern.c | 8 +-
11 files changed, 160 insertions(+), 209 deletions(-)
