Zvi Effron <zeffron@xxxxxxxxxxxxx> writes: > On Wed, Sep 22, 2021 at 1:01 PM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: >> >> Jakub Kicinski <kuba@xxxxxxxxxx> writes: >> >> > On Wed, 22 Sep 2021 00:20:19 +0200 Toke Høiland-Jørgensen wrote: >> >> >> Neither of those are desirable outcomes, I think; and if we add a >> >> >> separate "XDP multi-buff" switch, we might as well make it system-wide? >> >> > >> >> > If we have an internal flag 'this driver supports multi-buf xdp' cannot we >> >> > make xdp_redirect to linearize in case the packet is being redirected >> >> > to non multi-buf aware driver (potentially with corresponding non mb aware xdp >> >> > progs attached) from mb aware driver? >> >> >> >> Hmm, the assumption that XDP frames take up at most one page has been >> >> fundamental from the start of XDP. So what does linearise mean in this >> >> context? If we get a 9k packet, should we dynamically allocate a >> >> multi-page chunk of contiguous memory and copy the frame into that, or >> >> were you thinking something else? >> > >> > My $.02 would be to not care about redirect at all. >> > >> > It's not like the user experience with redirect is anywhere close >> > to amazing right now. Besides (with the exception of SW devices which >> > will likely gain mb support quickly) mixed-HW setups are very rare. >> > If the source of the redirect supports mb so will likely the target. >> >> It's not about device support it's about XDP program support: If I run >> an MB-aware XDP program on a physical interface and redirect the (MB) >> frame into a container, and there's an XDP program running inside that >> container that isn't MB-aware, bugs will ensue. Doesn't matter if the >> veth driver itself supports MB... >> >> We could leave that as a "don't do that, then" kind of thing, but that >> was what we were proposing (as the "do nothing" option) and got some >> pushback on, hence why we're having this conversation :) >> >> -Toke >> > > I hadn't even considered the case of redirecting to a veth pair on the same > system. I'm assuming from your statement that the buffers are passed directly > to the ingress inside the container and don't go through the sort of egress > process they would if leaving the system? And I'm assuming that's as an > optimization? Yeah, if we redirect an XDP frame to a veth, the peer will get the same xdp_frame, without ever building an SKB. > I'm not sure that makes a difference, though. It's not about whether the > driver's code is mb-capable, it's about whether the driver _as currently > configured_ could generate multiple buffers. If it can, then only an mb-aware > program should be able to be attached to it (and tail called from whatever's > attached to it). If it can't, then there should be no way to have multiple > buffers come to it. > > So in the situation you've described, either the veth driver should be in a > state where it coalesces the multiple buffers into one, fragmenting the frame > if necessary or drops the frame, or the program attached inside the container > would need to be mb-aware. I'm assuming with the veth driver as written, this > might mean that all programs attached to the veth driver would need to be > mb-aware, which is obviously undesirable. Hmm, I guess that as long as mb-frames only show up for large MTUs, the MTU of the veth device would be a limiting factor just like for physical devices, so we could just apply the same logic there. Not sure why I didn't consider that before :/ > All of which significantly adds to the complexity to support mb-aware, so maybe > this could be developed later? Initially we could have a sysctl toggling the > state 0 single-buffer only, 1 multibuffer allowed. Then later we _could_ add a > state for dynamic control once all XDP supporting drivers support the necessary > dynamic functionality (if ever). At that point we'd have actual experience with > the sysctl and could see how much of a burden having static control is. > > I may have been misinterpreting your use case though, and you were talking > about the XDP program running on the egress side of the redirect? Is that what > you were talking about case? No I was talking about exactly what you outlined above. Although longer term, I also think we can use XDP mb as a way to avoid having to linearise SKBs when running XDP on them in veth (and for generic XDP) :) -Toke
