On Thu, Feb 23, 2023 at 3:22 PM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: > > Stanislav Fomichev <sdf@xxxxxxxxxx> writes: > > > I'd like to discuss a potential follow up for the previous "XDP RX > > metadata" series [0]. > > > > Now that we can access (a subset of) packet metadata at RX, I'd like to > > explore the options where we can export some of that metadata on TX. And > > also whether it might be possible to access some of the TX completion > > metadata (things like TX timestamp). > > > > I'm currently trying to understand whether the same approach I've used > > on RX could work at TX. By May I plan to have a bunch of options laid > > out (currently considering XSK tx/compl programs and XDP tx/compl > > programs) so we have something to discuss. > > I've been looking at ways of getting a TX-completion hook for the XDP > queueing stuff as well. For that, I think it could work to just hook > into xdp_return_frame(), but if you want to access hardware metadata > it'll obviously have to be in the driver. A hook in the driver could > certainly be used for the queueing return as well, though, which may > help making it worth the trouble :) Yeah, I'd like to get to completion descriptors ideally; so nothing better than a driver hook comes to mind so far :-( (I'm eye-balling mlx5's mlx5e_free_xdpsq_desc AF_XDP path mostly so far). > > I'd like to some more input on whether applying the same idea on TX > > makes sense or not and whether there are any sensible alternatives. > > (IIRC, there was an attempt to do XDP on egress that went nowhere). > > I believe that stranded because it was deemed not feasible to cover the > SKB TX path as well, which means it can't be symmetrical to the RX hook. > So we ended up with the in-devmap hook instead. I'm not sure if that's > made easier by multi-buf XDP, so that may be worth revisiting. > > For the TX metadata you don't really have to care about the skb path, I > suppose, so that may not matter too much either. However, at least for > the in-kernel xdp_frame the TX path is pushed from the stack anyway, so > I'm not sure if it's worth having a separate hook in the driver (with > all the added complexity and overhead that entails) just to set > metadata? That could just as well be done on push from higher up the > stack; per-driver kfuncs could still be useful for this, though. > > And of course something would be needed so that that BPF programs can > process AF_XDP frames in the kernel before they hit the driver, but > again I'm not sure that needs to be a hook in the driver. Care to elaborate more on "push from higher up the stack"? I've been thinking about mostly two cases: - XDP_TX - I think this one technically doesn't need an extra hook; all metadata manipulations can be done at xdp_rx? (however, not sure how real that is, since the descriptors are probably not exposed over there?) - AF_XDP TX - this one needs something deep in the driver (due to tx zc) to populate the descriptors? - anything else? > In any case, the above is just my immediate brain dump (I've been > mulling these things over for a while in relation to the queueing > stuff), and I'd certainly welcome more discussion on the subject! :) Awesome, thanks for the dump! Will try to keep you in the loop! > -Toke
