On Wed, Jan 02, 2019 at 11:32:46AM -0800, Sagi Grimberg wrote: > This was directed to the proposal in the patch set, > ib_device->kverbs_provider is really not a great design choice for > and interface.. Yeah, it really isn't.. > > The saner thing to do is to use UDP and one of > > the high speed ethernet packet processing flavours available these > > days. > > AFAICT we still have a long way before an application can actually do > termination + zcopy with whats available today. Forwarding is more > reasonable, I agree. I thought the AF_XDP stuff was doing zcopy now? > > So what is your standard for determining if a device is part of the > > RDMA subsystem or not? > > I'm not sure, but its seems slightly odd to me that the vast majority > of RDMA use cases is probably not our ulps (as much as I'd like them to > be ;)) but we state them as the bar. Well, it is the vast majority of the in-kernel use cases, for sure :) > Well, I agree that UD is kind of a low bar, and probably most of the > value of the EFA device comes from their SRD transport. > > Is usnic a burdan is because its not actively maintained in a > subsystem that is constantly evolving, or because it implements a > small subset of an RDMA device functionality? I think because everytime someone wants to do something to refactor the core API's (like the completion stuff, or the RDMA WC stuff) they've looked into usnic to see if it would break and got all confused. For instance, EFA did not implement create_ah properly, so anyone looking at how AH works for kverbs will become very confused by it. I suppose this is why just blocking kverbs entirely has come up as a proposal. People working on kverbs *do not* need to care about non-kverbs drivers at all. Part of this is how uverbs and kverbs are really roughly pushed into the same driver API, so a driver can't just say it supports uverbs only... > Personally, given that most of RDMA usage lives in userland, I would > think that having a uverbs provider is a more appropriate bar than > supporting our kernel consumers. But I (like you and others) would be > more than happy seeing both supported. We don't really have a compliance test or anything for uverbs, beyond the stuff in rdma-core, so if a driver doesn't support that stuff there is no way to know if the device or driver is implementing verbs correctly.. > Both can be set as a bar, but one could argue that its an > unnecessarily high bar (if its user-base has no interest in running > our kernel consumers). So how should we support non-verbs things? EFA seems particularly difficult because it is a *little* verbs like, and does seem to fit into the uverbs system somewhat OK. > > The EFA device doesn't support rkeys: it *clearly* doesn't do the > > thing we call RDMA. > > A lot of applications don't use rkeys. We even have a kernel consumer > that don't use rkeys (9p) but still is using RDMA devices. It uses only SEND? > > > Isn't it enough that something like rsockets can run on a device to > > > justify its existence? > > > > ?? rsockets requires RC RDMA QPs, EFA won't support it. > > I was referring to datagram rsockets... Even so, I don't think EFA has an addressing model compatible with rsockets, it doesn't use RDMA-CM either, which I think rsockets requires still for UD?? Jason
