On Jul 24, 2015, at 3:10 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: > On Fri, Jul 24, 2015 at 01:46:05PM -0400, Chuck Lever wrote: >>> I'm not surprised since invalidate is sync. I belive you need to >>> incorporate SEND WITH INVALIDATE to substantially recover this >>> overhead. >> >> I tried to find another kernel ULP using SEND WITH INVALIDATE, but >> I didn’t see one. I assume you mean the NFS server would use this >> WR when replying, to knock down the RPC’s client MRs remotely? > > Yes. I think the issue with it not being used in the kernel is mainly > to do with lack of standardization. The verb cannot be used unless > both sides negotiate it and perhaps the older RDMA protocols have not > been revised to include that. And RPC-over-RDMA version 1 does not have any way to signal that the server has invalidated the MRs. Such signaling would be a pre-requisite to allow the Linux NFS/RDMA client to interoperate with non-Linux NFS/RDMA servers that do not have such support. >> Only the RPC/RDMA header has to be parsed, but yes. The needed >> parsing is handled in rpcrdma_reply_handler right before the >> .ro_unmap_unsync call. > > Right, okay, if this could be done in the rq callback itself rather > than bounce to a wq and immediately turn around the needed invalidate > posts you'd get back a little more overhead by reducing the time to > turn it around... Then bounce to the wq to complete from the SQ > callback ? For FRWR, you could post LINV from the receive completion upcall handler, and handle the rest of the invalidation from the send completion upcall, then poke the RPC reply handler. But this wouldn’t work at all for FMR, whose unmap verb is synchronous, would it? I’m not sure we’d buy more than a few microseconds here, and the receive upcall is single-threaded. I’ll move the “lost WC” discussion to another thread. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
