On 04-Jan-19 06:11, Jason Gunthorpe wrote: > On Thu, Jan 03, 2019 at 06:32:56PM -0500, Doug Ledford wrote: > >>> The original point is still un-addressed. We, as a community, have >>> identified usnic as a bad idea that does not belong in >>> drivers/infiniband. This has been a pretty much universal position of >>> everyone who has worked on the core RDMA stack. >> >> I'll generally agree with this. It turns out usnic is not so much about >> access to a shared device with PD, QP, CQ, MR and AH support, but an >> SRIOV network device attached to a process with minimal (but sufficient >> I think) security measures to at least make sure that process can't >> listen in on other traffic. From there, it's just UDP on a private >> network adapter. > > EFA looks the same to me, except instead of UDP it is doing a bit > more, but it still just datagram processing. There are no verbs here > other than SEND and RECV. There is no RMA.. > > The functions usnic and EFA stub to EOPNOTSUPP/null look super similar > to me. The stubs are there because of IB_MANDATORY_FUNC. > > The fake PD construction looks similar, except EFA uses something > outside the driver to create it. I wouldn't call it fake. Just because the PD isn't backed up by an object in the device doesn't mean it's not used by the device. We can pass alloc_pd through our command interface to the device, it's just not needed :). > > Neither support req_notify_cq, so they don't really support verbs CQ > semantics. I guess polling only? This one is lack of implementation, not lack of support. We can and will implement CQ notifications. > >>> Do we want to reverse course on that and accept that usnic-like things >>> are actually OK? (ie no kverbs, no libibverbs and totally proprietary >>> everything) >> >> Even if we say the above is true, there are other factors to consider. >> Unlike usnic, efa follows much more closely to the existing rdma device >> model. It's not a separate device per process, it's a single shared >> device. > > I've never used usnic, is this true? It sure looks like it can > allocate ucontexts dynamically?? I think that is what the vnic stuff > is for?? Maybe the ucontext limit is pretty small, but there sure is > alot of code here if the limit is just 1.. We also don't know the EFA > ucontext limit ... There is no ucontext limit. > >> It actually *does* have the concept of PD, CQ, QP, and AH. It >> doesn't do UDP, it actually does post_send and post_recv. > > I think it is the same, usnic once had a post_send/recv implementation > in it's libibverbs provider, pretty much anything that has a command > queue for datagram could implement basic (but maybe non-conforming) > post_send/recv verbs for UD. > > So if there is a EFA provider it would be in the same place as where > usnic started at before libfabric <shrug> > >> above makes me inclined to be more accepting of efa. If you add in a >> libibverbs provider and actual official, working UD support, then even >> more so. > > running through libibverbs DV and having enough to be an to do the > standard ud_ping_pong would be better. > > But, IIRC, usnic could do ud_ping_pong at one point too. > >>> verbs is supposed to be a multi-vendor standard, not an enumeration of >>> every kind of proprietary device-specific behavior. >> >> What it's supposed to be...and what it is...well, that's evolving on a >> day to day basis, and it's already got plenty of proprietary, device- >> specific behavior. > > It certainly does.. > >> To be honest, I don't even consider libibverbs a viable programming >> interface anymore for the most part. What I mean by this is that >> libibverbs exists, and there are already existing consumers of >> libibverbs. I expect those things to continue. But, for the most part, >> people are encouraged now a days to use another interface at a higher >> level that then uses libibverbs behind the scenes: libfabric, UCX, MPIs, >> existing low latency middlewares, etc. Nobody I know if is telling >> people to write new code to libibverbs. > > I'm still seeing/hearing that libibverbs is the goto choice for things > that are not HPC-style focused as all the alternatives are currently > unsuitable in some way. > > It is also still the only choice if you want heterogenous > interoperability. > > But libibverbs is a horrible mess of an API at this point.. > >> But by that same token, if we are the common, device file opening, >> kernel interacting layer, then we must also be the place where all of >> the proprietary vendor specific stuff is implemented. And we already >> are. I cite all of the recent Mellanox Direct Verbs stuff as exactly >> that. > > We definately have allowed a side-car of proprietary stuff. The qib > chardev started it, and the Mellanox DV/devx stuff is a more > integrated take on the exact same idea.. > >> It does raise the question of whether, to be consistent, we should >> have requested psm/psm2 also be part of rdma-core. > > I did ask, was told 'not interested' :) > >> Mellanox uses the driver direct model, Intel uses a separate cdev, >> and efa is apparently intending to follow Mellanox's example. I >> don't see a problem with that. > > Nor I.. > >>> Let alone the issue that libibverbs, assumes it has providers for all >>> the system devices and pukes if it doesn't - that will need fixing too >>> if we actually want to properly support this model. >> >> We can fix libibverbs if we ever do take a driver without a verbs >> driver, but I don't know that we are there yet, and I still think >> libibverbs as the shim to interface with the kernel is a good idea. > > Well we already have usnic without a verbs provider, everyone just > ignores the breakage it inflicts. > > I think that is a really bad place for a new driver to start out at.. > > Jason >
