On Thu, Mar 25, 2021 at 7:51 PM Konstantinos Kaffes <kkaffes@xxxxxxxxx> wrote: > > Great, thanks for the info! I will look into implementing this. > > For the time being, I implemented a version of my design with N^2 > sockets. I observed that when all traffic is directed to a single NIC > queue, the throughput is higher than when I use all N NIC queues. I am > using spinlocks to guard concurrent access to UMEM and the > fill/completion rings. When I use a single NIC queue, I achieve > ~1Mpps; when I use multiple ~550Kpps. Are these numbers reasonable, > and this bad scaling behavior expected? 1Mpps sounds reasonable with SKB mode. If you use something simple like the spinlock scheme you describe, then it will not scale. Check the sample xsk_fwd.c in samples/bpf in the Linux kernel repo. It has a mempool implementation that should scale better than the one you implemented. For anything remotely complicated, something that manages the buffers in the umem plus the fill and completion queues is usually required. This is called a mempool most of the time. User-space network libraries such as DPDK and VPP provide fast and scalable mempool implementations. It would be nice to add a simple one to libbpf, or rather libxdp as the AF_XDP functionality is moving over there. Several people have asked for it, but unfortunately I have not had the time. > > On Thu, 25 Mar 2021 at 00:24, Magnus Karlsson <magnus.karlsson@xxxxxxxxx> wrote: > > > > On Thu, Mar 25, 2021 at 7:25 AM Konstantinos Kaffes <kkaffes@xxxxxxxxx> wrote: > > > > > > Hello everyone, > > > > > > I want to write a multi-threaded AF_XDP server where all N threads can > > > read from all N NIC queues. In my design, each thread creates N AF_XDP > > > sockets, each associated with a different queue. I have the following > > > questions: > > > > > > 1. Do sockets associated with the same queue need to share their UMEM > > > area and fill and completion rings? > > > > Yes. In zero-copy mode this is natural since the NIC HW will DMA the > > packet into a umem that was decided long before the packet was even > > received. And this is of course before we even get to pick what socket > > it should go to. This restriction is currently carried over to > > copy-mode, however, in theory there is nothing preventing supporting > > multiple umems on the same netdev and queue id in copy-mode. It is > > just that nobody has implemented support for it. > > > > > 2. Will there be a single XSKMAP holding all N^2 sockets? If yes, what > > > happens if my XDP program redirects a packet to a socket that is > > > associated with a different NIC queue than the one in which the packet > > > arrived? > > > > You can have multiple XSKMAPs but you would in any case have to have > > N^2 sockets in total to be able to cover all cases. Sockets are tied > > to a specific netdev and queue id. If you try to redirect to socket > > with a queue id or netdev that the packet was not received on, it will > > be dropped. Again, for copy-mode, it would from a theoretical > > perspective be perfectly fine to redirect to another queue id and/or > > netdev since the packet is copied anyway. Maybe you want to add > > support for it :-). > > > > > I must mention that I am using the XDP skb mode with copies. > > > > > > Thank you in advance, > > > Kostis > > > > -- > Kostis Kaffes > PhD Student in Electrical Engineering > Stanford University
|