On Fri, Jan 04, 2019 at 07:45:59PM +0000, Hefty, Sean wrote: > > > Let's start by agreeing on what verbs is: an abstract definition of > > > the functionality that a NIC supports. > > > > No, I don't think that at all. Linux common verbs is a concrete > > realization of the the IBA and iWarp specifications - with well > > defined behaviors from the application layer right down to exactly > > which packets are formed on the wire. > > I was using definition provided by the IB spec, only replacing HCA > with NIC. IB spec is not Linux common verbs, and the use of 'abstract' in the IBA is not referring to some wide ranging mandate but an abstract interface description to support the very specific behaviors required in the spec. ie it is not defining an actual concrete C programming interface. > > The entire point of the verbs ecosystem was to enable heterogenous > > interoperability - by being extremely well specified. We now have > > many, many vendors implementing interoperable verbs to that spec. > > Nearly every single command carried across the uverbs ABI carries > device specific data. It's not generic at all. I was not speaking of uverbs, I was talking about kverbs and libibverbs. The ABI is not intended for any external consumption, nor is it imagined to be device agnostic. This is why uverbs has been so attractive to stuff like efa/usnic, they can stuff all the non-standard things into the device specific data and just ignore the common structures. However, from a subsystem perspective we are only allowing 'rdma' devices to use uverbs. I have been asking for some kind of consensus on what qualifies something to be a 'rdma' device. We've agreed on a high water mark (NVMEof) but what is the low watermark? > > > If we want to be more specific, does EFA support some subset of > > > the commands and data structures defined ib_user_verbs.h in a > > > manner that's similar to other devices? Can the uverbs driver > > > manage allocated resources (PDs, MRs, QPs, CQs, AHs) the same as > > > other devices. Based on patch 11 and responses to my questions, > > > it looks that way to me. There are a couple areas where the > > > functionality is missing (e.g. QP state transitions), but the > > > submission looks closer than not. > > > > I think you are stretching into your 'abstract verbs' idea to claim > > the objects are the same as other devices: > > No - I’m looking at the enum and structure definitions in > ib_user_verbs.h and seeing if they are used in a similar way. > I.e. what changes, if any, were needed to the uverbs driver to > support the calls? You are trying to say a bunch of apple seeds is the same as an apple - and I really disagree. IMHO, EFA clearly does not reach the minimum threshold to be called a RDMA common verbs device. The question is if we should set the subsystem threshold for 'uverbs access' to be below that (and how far below?). > > A CQ without notification is not really a CQ. > > Hard polling on a CQ is a valid programming model. Regardless, what > does it matter if a device responds to a > IB_USER_VERBS_CMD_REQ_NOTIFY_CQ with ENOSYS? Why should that be > considered unacceptable? In this case the only reason this ugly hack was put in the driver is because the common code would *REFUSE* to register the driver unless there is a req_notify_cq implementation (and others) because it is declared as a mandatory part of common verbs for decades now. The IBA has things marked MANDATORY for a reason - IMHO something that doesn't implement 'enough' of the mandatory statements no longer qualifies to be called verbs. Jason
