On Thu, Nov 24, 2022 at 4:36 PM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote:
>
> Maciej Fijalkowski <maciej.fijalkowski@xxxxxxxxx> writes:
>
> > On Thu, Nov 24, 2022 at 04:17:01PM +0100, Maciej Fijalkowski wrote:
> >> On Thu, Nov 24, 2022 at 03:39:20PM +0100, Toke Høiland-Jørgensen wrote:
> >> > Jakub Kicinski <kuba@xxxxxxxxxx> writes:
> >> >
> >> > > On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
> >> > >> > Good idea, prototyped below, lmk if it that's not what you had in mind.
> >> > >> >
> >> > >> > struct xdp_buff_xsk {
> >> > >> > struct xdp_buff xdp; /* 0 56 */
> >> > >> > u8 cb[16]; /* 56 16 */
> >> > >> > /* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
> >> > >>
> >> > >> As pahole helpfully says here, xdp_buff is actually only 8 bytes from
> >> > >> being a full cache line. I thought about adding a 'cb' field like this
> >> > >> to xdp_buff itself, but figured that since there's only room for a
> >> > >> single pointer, why not just add that and let the driver point it to
> >> > >> where it wants to store the extra context data?
> >> > >
> >> > > What if the driver wants to store multiple pointers or an integer or
> >> > > whatever else? The single pointer seems quite arbitrary and not
> >> > > strictly necessary.
> >> >
> >> > Well, then you allocate a separate struct and point to that? Like I did
> >> > in mlx5:
> >> >
> >> >
> >> > + struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
> >> > + struct xdp_buff xdp = { .drv_priv = &mlctx };
> >> >
> >> > but yeah, this does give an extra pointer deref on access. I'm not
> >> > really opposed to the cb field either, I just think it's a bit odd to
> >> > put it in struct xdp_buff_xsk; that basically requires the driver to
> >> > keep the layouts in sync.
> >> >
> >> > Instead, why not but a cb field into xdp_buff itself so it can be used
> >> > for both the XSK and the non-XSK paths? Then the driver can just
> >> > typecast the xdp_buff into its own struct that has whatever data it
> >> > wants in place of the cb field?
Agreed, maybe having an explicit cb field in the xdp_buff is a nice
compromise (assuming, over time, most devices will use it).
> >> Why can't you simply have a pointer to xdp_buff in driver specific
> >> xdp_buff container which would point to xdp_buff that is stack based (or
> >> whatever else memory that will back it up - I am about to push a change
> >> that makes ice driver embed xdp_buff within a struct that represents Rx
> >> ring) for XDP path and for ZC the pointer to xdp_buff that you get from
> >> xsk_buff_pool ? This would satisfy both sides I believe and would let us
> >> keep the same container struct.
> >>
> >> struct mlx4_xdp_buff {
> >> struct xdp_buff *xdp;
> >> struct mlx4_cqe *cqe;
> >> struct mlx4_en_dev *mdev;
> >> struct mlx4_en_rx_ring *ring;
> >> struct net_device *dev;
> >> };
> >
> > Nah this won't work from kfunc POV, probably no way to retrieve the
> > mlx4_xdp_buff based on xdp_buff ptr that needs to be used as an arg.
> >
> > Sorry I'll think more about it, in the meantime let's hear more voices
> > whether we should keep Stan's original approach + modify xdp_buff_xsk or
> > go with Toke's proposal.
>
> OK, so I played around with the mlx5 code a bit more, and I think the
> "wrapping struct + cb area" can be made to work without too many ugly
> casts; I'll send an updated version of the mlx5 patches with this
> incorporated tomorrow, after I've run some tests...
I'll probably send a v3 sometime tomorrow (PST), so maybe wait for me
to make sure we are working on the same base?
Or LMK if you prefer to do it differently..
