> On Jul 10, 2017, at 5:32 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: > > On Mon, Jul 10, 2017 at 05:29:53PM -0400, Chuck Lever wrote: >> >>> On Jul 10, 2017, at 5:24 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: >>> >>> On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote: >>> >>>>>> Or I could revert all the "map page cache pages" logic and >>>>>> just use memcpy for small NFS WRITEs, and RDMA the rest of >>>>>> the time. That keeps everything simple, but means large >>>>>> inline thresholds can't use send-in-place. >>>>> >>>>> Don't you have the same problem with RDMA WRITE? >>>> >>>> The server side initiates RDMA Writes. The final RDMA Write in a WR >>>> chain is signaled, but a subsequent Send completion is used to >>>> determine when the server may release resources used for the Writes. >>>> We're already doing it the slow way there, and there's no ^C hazard >>>> on the server. >>> >>> Wait, I guess I meant RDMA READ path. >>> >>> The same contraints apply to RKeys as inline send - you cannot DMA >>> unmap rkey memory until the rkey is invalidated at the HCA. >>> >>> So posting an invalidate SQE and then immediately unmapping the DMA >>> pages is bad too.. >>> >>> No matter how the data is transfered the unmapping must follow the >>> same HCA synchronous model.. DMA unmap must only be done from the send >>> completion handler (inline send or invalidate rkey), from the recv >>> completion handler (send with invalidate), or from QP error state teardown. >>> >>> Anything that does DMA memory unmap from another thread is very, very >>> suspect, eg async from a ctrl-c trigger event. >> >> 4.13 server side is converted to use the rdma_rw API for >> handling RDMA Read. For non-iWARP cases, it's using the >> local DMA key for Read sink buffers. For iWARP it should >> be using Read-with-invalidate (IIRC). > > The server sounds fine, how does the client work? The client does not initiate RDMA Read or Write today. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
