On 02-Jan-19 23:41, Sagi Grimberg wrote: > >>> AFAICT we still have a long way before an application can actually do >>> termination + zcopy with whats available today. Forwarding is more >>> reasonable, I agree. >> >> I thought the AF_XDP stuff was doing zcopy now? > > Apparently it does, on a intel nics for now... cool. > >>> Well, I agree that UD is kind of a low bar, and probably most of the >>> value of the EFA device comes from their SRD transport. >>> >>> Is usnic a burdan is because its not actively maintained in a >>> subsystem that is constantly evolving, or because it implements a >>> small subset of an RDMA device functionality? >> >> I think because everytime someone wants to do something to refactor >> the core API's (like the completion stuff, or the RDMA WC stuff) >> they've looked into usnic to see if it would break and got all >> confused. > > You're right, every time I swung by it, I didn't understand what that > thing is doing but assumed its broken so I didn't worry about it too > much... > >> For instance, EFA did not implement create_ah properly, so anyone >> looking at how AH works for kverbs will become very confused by it. > > To an extent it cannot be fixed? Are we talking about the fact that EFA's create_ah is not atomic? That was fixed using the sleepable flag. We can also make it atomic, but as discussed, it makes no sense to do that until we actually go through the atomic flows. > >> I suppose this is why just blocking kverbs entirely has come up as a >> proposal. People working on kverbs *do not* need to care about >> non-kverbs drivers at all. >> >> Part of this is how uverbs and kverbs are really roughly pushed into >> the same driver API, so a driver can't just say it supports uverbs >> only... > > Well, there are a few devices that we don't really know to support our > kernel consumers (or at least I've never heard of someone who verified > them), and some that are known to be broken/deprecated. The reason for > this is because no one uses them to run ulps, which is the case for EFA > most likely. Correct, that's the reasoning for the kverbs_provider flag. Jason, do you prefer Sagi's suggestion to make the drivers advertise their supported QP types instead of the kverbs provider interface? > >>> Personally, given that most of RDMA usage lives in userland, I would >>> think that having a uverbs provider is a more appropriate bar than >>> supporting our kernel consumers. But I (like you and others) would be >>> more than happy seeing both supported. >> >> We don't really have a compliance test or anything for uverbs, beyond >> the stuff in rdma-core, so if a driver doesn't support that stuff >> there is no way to know if the device or driver is implementing verbs >> correctly.. > > Sounds like a needed testing suite (even regardless of the discussion > here). > >>> Both can be set as a bar, but one could argue that its an >>> unnecessarily high bar (if its user-base has no interest in running >>> our kernel consumers). >> >> So how should we support non-verbs things? EFA seems particularly >> difficult because it is a *little* verbs like, and does seem to fit >> into the uverbs system somewhat OK. > > I didn't say support non-verbs things. I personally think that uverbs > interface is required (and I think Gal agreed to add one to his future > submissions). > > Anyways... its complicated I guess.. its hard to come up with a > reasonable bar half way in... Its just ones opinion.. > >>>> The EFA device doesn't support rkeys: it *clearly* doesn't do the >>>> thing we call RDMA. >>> >>> A lot of applications don't use rkeys. We even have a kernel consumer >>> that don't use rkeys (9p) but still is using RDMA devices. >> >> It uses only SEND? > > Yep. > >>>>> Isn't it enough that something like rsockets can run on a device to >>>>> justify its existence? >>>> >>>> ?? rsockets requires RC RDMA QPs, EFA won't support it. >>> >>> I was referring to datagram rsockets... >> >> Even so, I don't think EFA has an addressing model compatible with >> rsockets, it doesn't use RDMA-CM either, which I think rsockets >> requires still for UD?? > > I'd assume that EFA proprietary stuff matches IPv4/v6 to their > something so they can hook into ucma? > > otherwise how would addressing work at all? Perhaps Gal can share > more on this... Our addressing does not rely on rdmacm, also, there is no matching netdevice (ipv4/6) for the EFA ib device. Each EFA device has a 16 bytes opaque GID (queried from the device) that should be specified when creating the AH. libfabric's connection manager (out of band) is used to exchange these device GIDs and destination QP numbers. Does that answer your questions?
