On Fri, Dec 21, 2018 at 01:28:14AM +0000, Hefty, Sean wrote: > > > I agree. It has the general abstract hardware semantics that I > > > associated as 'verbs' - PD, MR, AH, QP, CQ. > > > > All most all modern high performance hardware has some variation on > > QP's and CQ's. > > I draw a distinction between command queues and QPs and whether the > addressing is closely coupled with it. It doesn't sound like SRD has addressing associated with the QP, as it is unconnected. Addressing is surely associated with the individual command? Whats the difference between a command queue and a verbs QP? Verbs QPs have the verbs defined properties and states. (notice EFA hardwired/ignored alot of these). It has the verbs state machines. QPs execute *verbs* actions (SEND/RECV/READ/WRITE/etc). Should 'rdma' be a subsystem that allows any arbitary process isolated command queues to be exposed to user space?? That is what USNIC is doing. But others have said no to this, they want to build a new subsystem more like vfio.. > > It doesn't have rkeys, right? So the PD&MR is just some variation of > > lkey gather/scatter from user memory?? > > The PD also controls the AH. Hmm? Not really in verbs. Is EFA doing something different? > You could argue that it's unnecessary to group QPs, MRs, and AHs by > PDs, but that does match verb semantics. The PD provides process > isolation. verbs semantics require the PD grouping because it is a security boundary for the rkey. The verbs user must have control over the PD to ensure that multiple remote connections in the same process have isolated rkeys from each other. If it wasn't for the rkey then all those objects would be simply grouped to the verbs FD with no PD. > > A MR without an rkey capability is not a MR, we've been calling that > > object a 'umem' in verbs lately. > > Well, the IB spec still calls this a MR. :b And AF_XDP calls it a umem and vfio calls it a 'dma map'. > > This is my objection to these drivers, the device does not implement > > anything close to the expected verbs semantics for any of the objects > > it seems to use. > > It looks similar to an IB device that only supports UD QPs. Gal already said it doesn't support 'verbs' UD. It supports two kinds of proprietary QPs, as far as I can tell. One is sort of like UD, maybe. > You honestly don't think you could write an app against SRD knowing > nothing more than you know now? More than *I* know? Certainly not! I assume you've been reviewing their libfabric driver, so I think you know alot more than I do.. .. and that isn't the standard for defining a verb extension anyhow. > Yes, the documentation could be improved to clarify the semantics, > but I do think a fair attempt could be made to program an app to use > SRD. The semantics would need to be documented to a level that another vendor could make a compatible implementation - at a *minimum*. .. and that ignores the issue that the wire protocol is proprietary, which we haven't ignored in the past. .. and that they have stated no intention to actually provide a verbs API for this functionality, only doing libfabric, it seems. So I see this the same as usnic. A non-verbs proprietary protocol that is used to build something more complicated in userspace. This is in contrast to QIB/HFI that used their non-verbs HW protocol to build verbs in SW, in the kernel. > > > I still view SRD as a generic QP type for reliable-unconnected send > > > and receive messages. > > > > Something proprietary and undocumented cannot be part of common > > verbs. I am absolutely adament on this. We cannot maintain the core > > code as a patchwork of undocumented driver-specific crap. > > I'm not sure why something proprietary is considered crap. That > seems unnecessarily judgmental. Have you looked at the usnic kernel driver? :( > The behavior of SRD can be described in more detail, even keeping > the protocol proprietary. Sure, but this is a line we have not yet crossed in RDMA. > > Mellanox already has a reliable delivery unconnected QP type that > > supports RDMA. It is exposed out of the driver as IBV_QPT_DRIVER > > because the protocol and behavior isn't documented. > > I would like to see this be defined generically as well, and really > anything that supports verbs in an 'intuitive' way. Generally someone should use UCX to access this protocol, not verbs.. Just like you should use (apparently) libfabric to access SRD, not verbs. I'm not sure who it benefits to document SRD just to justify being verbs to get the uapi when nothing in the kernel or userspace will use the 'SRD verbs'. > > I don't see any difference with EFA's proprietary protocol. > > I still don't see the point of having every driver use value 255 to > indicate that the real value is some other location. It indicates it is not supported by the core stack, and will not be supported by any other driver. Seeing QPT_DRIVER outside the driver/hw directory is a huge red flag that patches will be NAK'd - ie ULPs can't use QPT_DRIVER. It is the appropriate placement for something expected to be propriety. If there is an industry will to standardize then the definition can change. > We're not constrained on enum values, and this just makes it more > likely that a request can get forwarded to the wrong driver. ioctl has protections against this already. Look, this is a very strange situation. Since QIB/HFI PSM we have been allowing drivers to expose their own command channels to userspace that are proprietary. That isn't a problem, per say. The objection is using the RDMA subsystem to access the uapi channel when the device isn't an RDMA device and doesn't provide verbs ops toward the kernel. This has confused/complicated a lot of the kernel work because we see bad/confusing/non-standard implementations of verbs objects that don't act the way they should. In the past this kernel work has just said 'ignore usnic, it should even be here' - and honestly there is a bit of a "shrug, it is cisco's problem if usnic somehow breaks" because it is so incredibly far from all the other drivers. I said it the last time this came up, if we want to go in this direction we should consider that these devices are somehow entirely different from the verbs oriented stack, and maybe they don't abuse the kernel facing verbs APIs in their quest to get access to user space. But I haven't seen a proposal for something like this.. Even something fairly simple like declaring them a NON_VERBS device and hiding them from the in-kernel APIs might be enough. But them I'm not sure how we justify using the common verbs uAPI.. Mellanox and HFI both defined their own non-verbs uAPIs, maybe that is more appropriate here. It is hard to say when there is so little public information <shrug> Jason
