The socket is an endpoint, where I'd like data to end up (or get sent
from). If the kernel can attach the socket to a hardware queue,
there's zerocopy if not, copy-mode. Dito for Tx.
Well XDP programs per RXQ is just a building block to achieve this.
As Van Jacobson explain[4], sockets or applications "register" a
"transport signature", and gets back a "channel". In our case, the
netdev-global XDP program is our way to register/program these transport
signatures and redirect (e.g. into the AF_XDP socket).
This requires some work in software to parse and match transport
signatures to sockets. The XDP programs per RXQ is a way to get
hardware to perform this filtering for us.
[4] http://www.lemis.com/grog/Documentation/vj/lca06vj.pdf
There are a lot of things that are missing to build what you're
describing above. Yes, we need a better way to program the HW from
Linux userland (old topic); What I fail to see is how per-queue XDP is
a way to get hardware to perform filtering. Could you give a
longer/complete example (obviously with non-existing features :-)), so
I get a better view what you're aiming for?
Does a user (control plane) want/need to care about queues? Just
create a flow to a socket (out-of-band or inband) or to a netdevice
(out-of-band).
A userspace "control-plane" program, could hide the setup and use what
the system/hardware can provide of optimizations. VJ[4] e.g. suggest
that the "listen" socket first register the transport signature (with
the driver) on "accept()". If the HW supports DPDK-rte_flow API we
can register a 5-tuple (or create TC-HW rules) and load our
"transport-signature" XDP prog on the queue number we choose. If not,
when our netdev-global XDP prog need a hash-table with 5-tuple and do
5-tuple parsing.
Creating netdevices via HW filter into queues is an interesting idea.
DPDK have an example here[5], on how to per flow (via ethtool filter
setup even!) send packets to queues, that endup in SRIOV devices.
[5] https://doc.dpdk.org/guides/howto/flow_bifurcation.html
Do we envison any other uses for per-queue XDP other than AF_XDP? If
not, it would make *more* sense to attach the XDP program to the
socket (e.g. if the endpoint would like to use kernel data structures
via XDP).
As demonstrated in [5] you can use (ethtool) hardware filters to
redirect packets into VFs (Virtual Functions).
I also want us to extend XDP to allow for redirect from a PF (Physical
Function) into a VF (Virtual Function). First the netdev-global
XDP-prog need to support this (maybe extend xdp_rxq_info with PF + VF
info). Next configure HW filter to queue# and load XDP prog on that
queue# that only "redirect" to a single VF. Now if driver+HW supports
it, it can "eliminate" the per-queue XDP-prog and do everything in HW.
Again, let's try to be more concrete! So, one (non-existing) mechanism
to program filtering to HW queues, and then attaching a per-queue
program to that HW queue, which can in some cases be elided? I'm not
opposing the idea of per-queue, I'm just trying to figure out
*exactly* what we're aiming for.
My concern is, again, mainly that is a queue abstraction something
we'd like to introduce to userland. It's not there (well, no really
:-)) today. And from an AF_XDP userland perspective that's painful.
"Oh, you need to fix your RSS hashing/flow." E.g. if I read what
Jonathan is looking for, it's more of something like what Jiri Pirko
suggested in [1] (slide 9, 10).
Hey, maybe I just need to see the fuller picture. :-) AF_XDP is too
tricky to use from XDP IMO. Per-queue XDP program would *optimize*
AF_XDP, but not solving the filtering. Maybe starting in the
filtering/metadata offload path end of things, and then see what we're
missing.
If we'd like to slice a netdevice into multiple queues. Isn't macvlan
or similar *virtual* netdevices a better path, instead of introducing
yet another abstraction?
XDP redirect a more generic abstraction that allow us to implement
macvlan. Except macvlan driver is missing ndo_xdp_xmit. Again first I
write this as global-netdev XDP-prog, that does a lookup in a BPF-map.
Next I configure HW filters that match the MAC-addr into a queue# and
attach simpler XDP-prog to queue#, that redirect into macvlan device.
Just for context; I was thinking something like macvlan with
ndo_dfwd_add/del_station functionality. "A virtual interface that is
simply is a view of a physical". A per-queue program would then mean
"create a netdev for that queue".
