(Answered inline, below) On 29/09/2022 15.16, Marcus Wichelmann wrote:
Am 28.09.22 um 20:07 schrieb Jesper Dangaard Brouer:On 28/09/2022 15.54, Marcus Wichelmann wrote:I'm working for a cloud hosting provider and we're working on a new XDP-based networking stack for our VM-Hosts that uses XDP to accelerate the connectivity of our qemu/KVM VMs to the outside.Welcome to the community!Thank you!Sounds like an excellent use-case and opportunity for speeding up the RX packets from physical NIC into the VM. Good to hear someone (again) having this use-case. I've personally not been focused on this use-case lately, mostly because community members that I was interacting with changed jobs, away from cloud hosting companies. Good to have a user back in this area!Good to hear! Also, we'll probably not be the last ones coming up with this use-case. ;)
Yes, and remember to look at the effort done by people before... I urge you to read David Ahern's slides: https://legacy.netdevconf.info/0x14/pub/slides/24/netdev-0x14-XDP-and-the-cloud.pdf It is a details step-by-step explanation of your use-case, along with the pitfalls and gotchas. If you hit an issue, do remember to bring it to the attention of the community (e.g. xdp-newbies), then lurking kernel engineers likely will get motivated to fix these issues upstream.(Like slides explain improvements for redirects in kernel v5.4 + v5.6 + v5.8)
For this, we use XDP_REDIRECT to forward packets between the physical host NIC and the VM tap devices. The main issue we have now is, that our VM guests have some virtio NIC offloads enabled: rx/tx checksumming, TSO/GSO, GRO and Scatter-Gather.Supporting RX-checksumming is part of the plans for XDP-hints, although virtio_net is not part of my initial patchset.Great!
It should be trivial to add to virtio_net.
XDP-redirect with GRO and Scatter-Gather frames are part of the multi-buff effort (Cc Lorenzo), but currently XDP_REDIRECT with multi-buff is disabled (except for cpumap), because the lack of XDP-feature bits, meaning we cannot determine (in kernel) if receiving net_device supports multi-buff (Cc Kumar).Can this also be solved with XDP-Hints or is this an unrelated issue?
This is unrelated to XDP-hints.
The XDP multi-buffer support needed for TSO/GSO seems to be mostly thereA subtle detail is that both XDP-hints and XDP multi-buff are needed to get GRO/GSO kernel infra working. For the kernel to construct GRO-SKB based packets on XDP-redirected incoming xdp_frame's, the kernel code requires both RX-csum and RX-hash before coalescing GRO frames.already, but, to our understanding, the last missing part for full TSO/GSO support is a way to tell the physical NIC to perform the TSO/GSO offload.The TSO/GSO side is usually the TX side. The VM should be able to send out normal TSO/GSO (multi-buffer) packets.Currently the VM sends out multi-buffer packets, but after redirecting them, they are probably not getting segmented on the way out of the physical NIC. Or, as you wrote earlier, the XDP multi-buffer support isn't even used there and the packet just gets truncated on the way into XDP. I've not exactly traced that down yet, but you probably know better than me what's happening there.
XDP program on tap-device will likely cause drops of multi-buffer packets (send out by VM).
(1) First of all this XDP-tap program need to use the newer XDP program sub-type that known about multi-buffer packets.
(2) I'm not sure XDP-tap (virio_net) got multi-buffer support. Lorenzo or Jason do you know?
Because of that, the TX side offloads are more critical to us because we cannot easily disable them in the VMs. The RX side is less of an issue, because we have control over the physical NIC configuration and could temporarily disable all offloads there, until XDP supports them (which would of course be better). So RX offloads are very nice to have, but missing TX offloads are a show-stopper for this use-case, if we don't find a way to disable the offloads on all customer VMs.> Or are you saying this also gets disabled when enabling XDP on the > virtio_net RX side? I'm not sure what you mean with that. What gets disabled?
See Ahern's slide "Redirecting VM Egress Traffic". The libvirt config (or Qemu/kvm params) currently need to disables many of the offloads for XDP-on-tap to work. IMHO this is something we kernel developers need to fix/improve. (Cc Jason + Lorenzo)
I've seen the latest LPC 2022 talk from Jesper Dangaard Brouer regarding the planned XDP-Hints feature. But this was mainly about Checksum and VLAN offloads. Is supporting TSO/GSO also one of the goals you have in mind with these XDP-Hints proposals?As mentioned TSO/GSO is TX side. We (Cc Magnus) also want to extend XDP-hints to TX-side, to allow asking the HW to perform different offloads. Lets land RX-side first.Makes sense, thanks for clarifying your roadmap!
For your own roadmap, waiting for "TX-XDP-hints" is likely problematic. Thus, I would likely recommend NOT XDP-redirecting (TCP) traffic coming from the VMs, which will hit the XDP-tap BPF program. The XDP-tap program could selectively XDP-redirect the UDP packets (if your measurements show it to be faster). Start with XDP redirecting from the physical NIC device into the VMs. The XDP-hints coming from physical NIC device should be trivially to convert into the format KVM needs. Looking at kernel code we need to populate struct virtio_net_hdr (which is inside struct tun_xdp_hdr).
The "XDP Cloud-Provider" project page describes a very similar use-case to what we plan to do. What's the goal of this project?Yes, this sounds VERY similar to your use-case. I think you are referring to this: [1] https://xdp-project.net/areas/xdp-cloud-provider.html [2] https://github.com/xdp-project/xdp-cloudThe GitHub Link is a 404. Maybe this repository is private-only?
Yes, sorry about that git repo is marked private, because the project didn't take off.
We had two Cloud Hosting companies interested in this use-case and started a "sub" xdp-project, with the intent of working together on code[2] that implements concrete BPF tools, that functions as building blocks that the individual companies can integrate into their systems, separating out customer provisioning to the companies. (p.s. this approach have worked well for xdp-cpumap-tc[3] scaling tool)I wonder what these common building blocks could be. I think this would be mostly just a program that calls XDP-Redirect and also some XDP-Hints handling in the future. This could also be demonstrated as an example program.
Sure. I recommend you start with coding an eBPF example program, and if you want my help please base it on https://github.com/xdp-project/bpf-examples
While looking at our current XDP-Stack design draft, I think everything beyond that is highly specific to how the network infrastructure of the cloud hosting environment is designed and will probably be hard to apply to the requirements of other providers.
Hmm... I kind of disagree, but that should not stop you. I still encourage to decouple customer/VM provisioning in your design.
But of course, having a simple reference implementation of a XDP datapath that demonstrates how XDP can be used to connect VMs to the outside, would still be very useful. For documentation purposes, maybe not su much as a framework.
Great, lets start with PoC/MVP as sub-dir under: https://github.com/xdp-project/bpf-examples If we can iterate over a public 'xdp-cloud' bpf-example, then the community can easier reproduce the issues that devel process brings up. --Jesper
|