On Thu, Jan 03, 2019 at 08:14:54PM +0000, Hefty, Sean wrote: > > I assume that NVMe over fabric users/developers will disagree with > > you about such primary/secondary separation. > > The point both Sagi and Doug are making are valid. The primary > users of hardware in the linux-rdma subsystem are user space > applications. That is only true in certain verticals, particularly HPC. More generally there are lots of verticals that care about RDMA only to run kernel-based things like NVMeoF or SMB-Direct (windows) and never run a user applcation. Arguably there actually are more customers by number in the latter (though possibly more CPU cores/adaptors in the former) The original point is still un-addressed. We, as a community, have identified usnic as a bad idea that does not belong in drivers/infiniband. This has been a pretty much universal position of everyone who has worked on the core RDMA stack. Do we want to reverse course on that and accept that usnic-like things are actually OK? (ie no kverbs, no libibverbs and totally proprietary everything) I still haven't actually yet heard people saying yes to the above.. The discussion about kernel support started because I said supporting the existing RC ULPs *clearly* means the devices is not a 'usnic' class driver. Nobody has presented another criteria that makes EFA and usnic any different, so we are back to the start. Was usnic a bad idea or not? How do we support 'usnic' style devices without wrecking the rest of the stack? verbs is supposed to be a multi-vendor standard, not an enumeration of every kind of proprietary device-specific behavior. I'm kind of half wondering if the idea to hide 'usnic' devices from the kernel ULPs doesn't go far enough - maybe we should have a /dev/urdma for them as well? That would also go a long way to address some of my past annoyance that hfi/qib have a private cdev.. > That's why usNIC doesn't care about kernel users. That's why EFA > has no kernel support. I'd say they don't care because they aren't actually classic RDMA devices, they are something new. > Requiring support for a specific user space library is basically > pointless and unenforceable. Once the kABI is exposed, any software > can write to it. It is probably a bad idea to implement the uverbs abi without copying the new machinery for doing so from verbs, it is complicted, and the ioctl support makes it even more tricky. Let alone the issue that libibverbs, assumes it has providers for all the system devices and pukes if it doesn't - that will need fixing too if we actually want to properly support this model. Jason
