> On Jun 27, 2017, at 12:07 PM, Sagi Grimberg <sagi@xxxxxxxxxxx> wrote: > > >> Go back and browse the end of the thread: there's no need to change >> xprtrdma, and maybe no need to change the others either. > > I think there is, even with inline, xprtrdma dma maps the immediate > buffers, also the message head and tail so unmapping these buffers > without waiting for the send completion would trigger a IOMMU access > error (the HCA (re)tries to access an already unmapped buffer). Thinking out loud: IIRC the message head and tail reside in the persistently registered and DMA mapped buffers with few exceptions. However, when page cache pages are involved, xprtrdma will do a DMA unmap as you say. So: - we don't have a problem transmitting a garbled request thanks to exactly-once receive semantics - we don't have a problem with the timing of registration and invalidation on the initiator because the PD's DMA lkey is used - we do have a problem with DMA unmap Using only persistently registered and DMA mapped Send buffers should avoid the need to signal all Sends. However, when page cache pages are involved, then the Send needs to be signaled, and the pages unmapped only after Send completion, to be completely safe. Or, the initiator should just use RDMA Read and Write, and stick with small inline sizes. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
