On Sun, Dec 23, 2018 at 04:48:36PM +0200, Gal Pressman wrote: > On 21-Dec-18 05:15, Jason Gunthorpe wrote: > > On Fri, Dec 21, 2018 at 01:28:14AM +0000, Hefty, Sean wrote: > >>>> I agree. It has the general abstract hardware semantics that I > >>>> associated as 'verbs' - PD, MR, AH, QP, CQ. > >>> > >>> All most all modern high performance hardware has some variation on > >>> QP's and CQ's. > >> > >> I draw a distinction between command queues and QPs and whether the > >> addressing is closely coupled with it. > > > > It doesn't sound like SRD has addressing associated with the QP, as it > > is unconnected. Addressing is surely associated with the individual > > command? > > > > Whats the difference between a command queue and a verbs QP? Verbs QPs > > have the verbs defined properties and states. (notice EFA > > hardwired/ignored alot of these). It has the verbs state machines. QPs > > execute *verbs* actions (SEND/RECV/READ/WRITE/etc). > > EFA queues are not command queues, these are verbs queues that > support SEND/RECV operations using AH and QP numbers for addressing > and MR lkeys for local buffer access. We are missing the QP states > at the moment, but support for these will be added in the firmware. verbs QP's are command queues.. The question is if the EFA command queue does even verbs to be a verbs QP. If the device can implement IBTA UD QP's then probably yes. > > .. and that ignores the issue that the wire protocol is proprietary, > > which we haven't ignored in the past. > > I get your concerns, but how does the lack of documentation for the > wire protocol in EC2 network impact our users? What information that > affects EFA's usage is missing? Every customer visible feature will > be documented. It is not something we've done before, so there is no canned answer or precedent. > > So I see this the same as usnic. A non-verbs proprietary protocol that > > is used to build something more complicated in userspace. > > I'm not familiar with what usnic are doing, but this is not what EFA > is about. This will be more clear in our userspace bits. It is doing what this first EFA driver was doing :) Almost exactly, right down to not having a libibverbs provider. Except it used a different approach to creating umem.. > >> The behavior of SRD can be described in more detail, even keeping > >> the protocol proprietary. > > > > Sure, but this is a line we have not yet crossed in RDMA. > > Does proprietary implementation mean proprietary protocol? Proprietary protocol means the wire protocol is not documented. > > I'm not sure who it benefits to document SRD just to justify being > > verbs to get the uapi when nothing in the kernel or userspace will use > > the 'SRD verbs'. > > How is SRD different than DC? We'll expose a direct verb to > introduce the new QP type. DC isn't documented at all :) Otherwise it looks the pretty much the same to me. > If the specification for SRD will suffice, we can implement it > through standard ibv_create_qp. We are trying not to accumulate endless one-off implementations in verbs. If the implementation is expected to be a one-off then the appropriate place is in the DV side of the verbs world. > Also, we can implement a UD API for the kernel, it's not a smart > thing to do since there are no kernel users for that, but that would > be providing verbs ops towards the kernel and clear the non-standard > implementation suspicion. I wouldn't recommend adding dead code. Jason
