AFAICT we still have a long way before an application can actually do
termination + zcopy with whats available today. Forwarding is more
reasonable, I agree.
I thought the AF_XDP stuff was doing zcopy now?
Apparently it does, on a intel nics for now... cool.
Well, I agree that UD is kind of a low bar, and probably most of the
value of the EFA device comes from their SRD transport.
Is usnic a burdan is because its not actively maintained in a
subsystem that is constantly evolving, or because it implements a
small subset of an RDMA device functionality?
I think because everytime someone wants to do something to refactor
the core API's (like the completion stuff, or the RDMA WC stuff)
they've looked into usnic to see if it would break and got all
confused.
You're right, every time I swung by it, I didn't understand what that
thing is doing but assumed its broken so I didn't worry about it too
much...
For instance, EFA did not implement create_ah properly, so anyone
looking at how AH works for kverbs will become very confused by it.
To an extent it cannot be fixed?
I suppose this is why just blocking kverbs entirely has come up as a
proposal. People working on kverbs *do not* need to care about
non-kverbs drivers at all.
Part of this is how uverbs and kverbs are really roughly pushed into
the same driver API, so a driver can't just say it supports uverbs
only...
Well, there are a few devices that we don't really know to support our
kernel consumers (or at least I've never heard of someone who verified
them), and some that are known to be broken/deprecated. The reason for
this is because no one uses them to run ulps, which is the case for EFA
most likely.
Personally, given that most of RDMA usage lives in userland, I would
think that having a uverbs provider is a more appropriate bar than
supporting our kernel consumers. But I (like you and others) would be
more than happy seeing both supported.
We don't really have a compliance test or anything for uverbs, beyond
the stuff in rdma-core, so if a driver doesn't support that stuff
there is no way to know if the device or driver is implementing verbs
correctly..
Sounds like a needed testing suite (even regardless of the discussion
here).
Both can be set as a bar, but one could argue that its an
unnecessarily high bar (if its user-base has no interest in running
our kernel consumers).
So how should we support non-verbs things? EFA seems particularly
difficult because it is a *little* verbs like, and does seem to fit
into the uverbs system somewhat OK.
I didn't say support non-verbs things. I personally think that uverbs
interface is required (and I think Gal agreed to add one to his future
submissions).
Anyways... its complicated I guess.. its hard to come up with a
reasonable bar half way in... Its just ones opinion..
The EFA device doesn't support rkeys: it *clearly* doesn't do the
thing we call RDMA.
A lot of applications don't use rkeys. We even have a kernel consumer
that don't use rkeys (9p) but still is using RDMA devices.
It uses only SEND?
Yep.
Isn't it enough that something like rsockets can run on a device to
justify its existence?
?? rsockets requires RC RDMA QPs, EFA won't support it.
I was referring to datagram rsockets...
Even so, I don't think EFA has an addressing model compatible with
rsockets, it doesn't use RDMA-CM either, which I think rsockets
requires still for UD??
I'd assume that EFA proprietary stuff matches IPv4/v6 to their
something so they can hook into ucma?
otherwise how would addressing work at all? Perhaps Gal can share
more on this...
