> > I agree. It has the general abstract hardware semantics that I > > associated as 'verbs' - PD, MR, AH, QP, CQ. > > All most all modern high performance hardware has some variation on > QP's and CQ's. I draw a distinction between command queues and QPs and whether the addressing is closely coupled with it. > I'm not sure exactly what a device that can't do RDMA is doing with > MRs. It is still required for pinning memory buffers for send/receive operations. > It doesn't have rkeys, right? So the PD&MR is just some variation of > lkey gather/scatter from user memory?? The PD also controls the AH. You could argue that it's unnecessary to group QPs, MRs, and AHs by PDs, but that does match verb semantics. The PD provides process isolation. > A MR without an rkey capability is not a MR, we've been calling that > object a 'umem' in verbs lately. Well, the IB spec still calls this a MR. :b > This is my objection to these drivers, the device does not implement > anything close to the expected verbs semantics for any of the objects > it seems to use. It looks similar to an IB device that only supports UD QPs. It doesn't give the impression as being so far off from verbs as to be something completely unrelated. You honestly don't think you could write an app against SRD knowing nothing more than you know now? Yes, the documentation could be improved to clarify the semantics, but I do think a fair attempt could be made to program an app to use SRD. > > IMO, IB hardware semantic quirks should not carry over to more > > generic QP type definitions. EFA and IB have different addressing > > formats, so an app can't just switch between one UD QP and another. > > The QP type must specify broadly how the WR's and WC's work SRD appears to do this at a broad level. What additional semantics are you wanting to know? > > I still view SRD as a generic QP type for reliable-unconnected send > > and receive messages. > > Something proprietary and undocumented cannot be part of common > verbs. I am absolutely adament on this. We cannot maintain the core > code as a patchwork of undocumented driver-specific crap. I'm not sure why something proprietary is considered crap. That seems unnecessarily judgmental. The behavior of SRD can be described in more detail, even keeping the protocol proprietary. > Mellanox already has a reliable delivery unconnected QP type that > supports RDMA. It is exposed out of the driver as IBV_QPT_DRIVER > because the protocol and behavior isn't documented. I would like to see this be defined generically as well, and really anything that supports verbs in an 'intuitive' way. > I don't see any difference with EFA's proprietary protocol. I still don't see the point of having every driver use value 255 to indicate that the real value is some other location. We're not constrained on enum values, and this just makes it more likely that a request can get forwarded to the wrong driver. For me, this is a separate issue than defining SRD in such a way that it can be used generically outside of EFA. - Sean
