Great, thanks for the info! I will look into implementing this. For the time being, I implemented a version of my design with N^2 sockets. I observed that when all traffic is directed to a single NIC queue, the throughput is higher than when I use all N NIC queues. I am using spinlocks to guard concurrent access to UMEM and the fill/completion rings. When I use a single NIC queue, I achieve ~1Mpps; when I use multiple ~550Kpps. Are these numbers reasonable, and this bad scaling behavior expected? On Thu, 25 Mar 2021 at 00:24, Magnus Karlsson <magnus.karlsson@xxxxxxxxx> wrote: > > On Thu, Mar 25, 2021 at 7:25 AM Konstantinos Kaffes <kkaffes@xxxxxxxxx> wrote: > > > > Hello everyone, > > > > I want to write a multi-threaded AF_XDP server where all N threads can > > read from all N NIC queues. In my design, each thread creates N AF_XDP > > sockets, each associated with a different queue. I have the following > > questions: > > > > 1. Do sockets associated with the same queue need to share their UMEM > > area and fill and completion rings? > > Yes. In zero-copy mode this is natural since the NIC HW will DMA the > packet into a umem that was decided long before the packet was even > received. And this is of course before we even get to pick what socket > it should go to. This restriction is currently carried over to > copy-mode, however, in theory there is nothing preventing supporting > multiple umems on the same netdev and queue id in copy-mode. It is > just that nobody has implemented support for it. > > > 2. Will there be a single XSKMAP holding all N^2 sockets? If yes, what > > happens if my XDP program redirects a packet to a socket that is > > associated with a different NIC queue than the one in which the packet > > arrived? > > You can have multiple XSKMAPs but you would in any case have to have > N^2 sockets in total to be able to cover all cases. Sockets are tied > to a specific netdev and queue id. If you try to redirect to socket > with a queue id or netdev that the packet was not received on, it will > be dropped. Again, for copy-mode, it would from a theoretical > perspective be perfectly fine to redirect to another queue id and/or > netdev since the packet is copied anyway. Maybe you want to add > support for it :-). > > > I must mention that I am using the XDP skb mode with copies. > > > > Thank you in advance, > > Kostis -- Kostis Kaffes PhD Student in Electrical Engineering Stanford University
|