On Thu, Jan 03, 2019 at 06:32:56PM -0500, Doug Ledford wrote: > > The original point is still un-addressed. We, as a community, have > > identified usnic as a bad idea that does not belong in > > drivers/infiniband. This has been a pretty much universal position of > > everyone who has worked on the core RDMA stack. > > I'll generally agree with this. It turns out usnic is not so much about > access to a shared device with PD, QP, CQ, MR and AH support, but an > SRIOV network device attached to a process with minimal (but sufficient > I think) security measures to at least make sure that process can't > listen in on other traffic. From there, it's just UDP on a private > network adapter. EFA looks the same to me, except instead of UDP it is doing a bit more, but it still just datagram processing. There are no verbs here other than SEND and RECV. There is no RMA.. The functions usnic and EFA stub to EOPNOTSUPP/null look super similar to me. The fake PD construction looks similar, except EFA uses something outside the driver to create it. Neither support req_notify_cq, so they don't really support verbs CQ semantics. I guess polling only? > > Do we want to reverse course on that and accept that usnic-like things > > are actually OK? (ie no kverbs, no libibverbs and totally proprietary > > everything) > > Even if we say the above is true, there are other factors to consider. > Unlike usnic, efa follows much more closely to the existing rdma device > model. It's not a separate device per process, it's a single shared > device. I've never used usnic, is this true? It sure looks like it can allocate ucontexts dynamically?? I think that is what the vnic stuff is for?? Maybe the ucontext limit is pretty small, but there sure is alot of code here if the limit is just 1.. We also don't know the EFA ucontext limit ... > It actually *does* have the concept of PD, CQ, QP, and AH. It > doesn't do UDP, it actually does post_send and post_recv. I think it is the same, usnic once had a post_send/recv implementation in it's libibverbs provider, pretty much anything that has a command queue for datagram could implement basic (but maybe non-conforming) post_send/recv verbs for UD. So if there is a EFA provider it would be in the same place as where usnic started at before libfabric <shrug> > above makes me inclined to be more accepting of efa. If you add in a > libibverbs provider and actual official, working UD support, then even > more so. running through libibverbs DV and having enough to be an to do the standard ud_ping_pong would be better. But, IIRC, usnic could do ud_ping_pong at one point too. > > verbs is supposed to be a multi-vendor standard, not an enumeration of > > every kind of proprietary device-specific behavior. > > What it's supposed to be...and what it is...well, that's evolving on a > day to day basis, and it's already got plenty of proprietary, device- > specific behavior. It certainly does.. > To be honest, I don't even consider libibverbs a viable programming > interface anymore for the most part. What I mean by this is that > libibverbs exists, and there are already existing consumers of > libibverbs. I expect those things to continue. But, for the most part, > people are encouraged now a days to use another interface at a higher > level that then uses libibverbs behind the scenes: libfabric, UCX, MPIs, > existing low latency middlewares, etc. Nobody I know if is telling > people to write new code to libibverbs. I'm still seeing/hearing that libibverbs is the goto choice for things that are not HPC-style focused as all the alternatives are currently unsuitable in some way. It is also still the only choice if you want heterogenous interoperability. But libibverbs is a horrible mess of an API at this point.. > But by that same token, if we are the common, device file opening, > kernel interacting layer, then we must also be the place where all of > the proprietary vendor specific stuff is implemented. And we already > are. I cite all of the recent Mellanox Direct Verbs stuff as exactly > that. We definately have allowed a side-car of proprietary stuff. The qib chardev started it, and the Mellanox DV/devx stuff is a more integrated take on the exact same idea.. > It does raise the question of whether, to be consistent, we should > have requested psm/psm2 also be part of rdma-core. I did ask, was told 'not interested' :) > Mellanox uses the driver direct model, Intel uses a separate cdev, > and efa is apparently intending to follow Mellanox's example. I > don't see a problem with that. Nor I.. > > Let alone the issue that libibverbs, assumes it has providers for all > > the system devices and pukes if it doesn't - that will need fixing too > > if we actually want to properly support this model. > > We can fix libibverbs if we ever do take a driver without a verbs > driver, but I don't know that we are there yet, and I still think > libibverbs as the shim to interface with the kernel is a good idea. Well we already have usnic without a verbs provider, everyone just ignores the breakage it inflicts. I think that is a really bad place for a new driver to start out at.. Jason
