On Thu, Dec 20, 2018 at 10:28:41PM +0000, Hefty, Sean wrote: > > I followed the comments and your concerns and I'll try to address them all: > > Let me start by making clear that EFA is not an infiniband device, nor it > > aspires to being one, but I think it does fit the verbs model. > > I agree. It has the general abstract hardware semantics that I > associated as 'verbs' - PD, MR, AH, QP, CQ. All most all modern high performance hardware has some variation on QP's and CQ's. I'm not sure exactly what a device that can't do RDMA is doing with MRs. It doesn't have rkeys, right? So the PD&MR is just some variation of lkey gather/scatter from user memory?? A MR without an rkey capability is not a MR, we've been calling that object a 'umem' in verbs lately. This is my objection to these drivers, the device does not implement anything close to the expected verbs semantics for any of the objects it seems to use. What value is a PD if there are no rkeys? The only real purpose of a PD is to provide security via rkeys groupings - so why EFA does it make a fake PD? To forcibly fit into verbs, it seems. > > We can implement an rdma-core (libibverbs) userspace provider with > > support for standard UD (including the 40 bytes offset) and SRD > > QPs through direct verbs. I'll also add documentation for SRD QP > > type, even if we end up using it as a driver QP type. > > IMO, IB hardware semantic quirks should not carry over to more > generic QP type definitions. EFA and IB have different addressing > formats, so an app can't just switch between one UD QP and another. The QP type must specify broadly how the WR's and WC's work anything else is madness. Each protocol doesn't get to overload and redefine the existing well standardized definitions. If it doesn't conform the the WC and WR model from IBTA it does not get to use the IBV_QPT_UD/RC/UC/RD's values. Thats verbs. Even iwarp puts in the 40 byte header and doesn't need it. > I still view SRD as a generic QP type for reliable-unconnected send > and receive messages. Something proprietary and undocumented cannot be part of common verbs. I am absolutely adament on this. We cannot maintain the core code as a patchwork of undocumented driver-specific crap. Driver crap goes in drivers, where it is the driver's problem to maintain and keep working. > Along this line, given the constraints of the existing APIs, if a > reliable-unconnected QP that supported RDMA read/write were ever > introduced, I would also add that as a new QP type. Mellanox already has a reliable delivery unconnected QP type that supports RDMA. It is exposed out of the driver as IBV_QPT_DRIVER because the protocol and behavior isn't documented. I don't see any difference with EFA's proprietary protocol. Jason
