> On Jun 28, 2017, at 3:08 AM, Sagi Grimberg <sagi@xxxxxxxxxxx> wrote: > > >> Thinking out loud: >> IIRC the message head and tail reside in the persistently registered >> and DMA mapped buffers with few exceptions. >> However, when page cache pages are involved, xprtrdma will do a DMA >> unmap as you say. >> So: >> - we don't have a problem transmitting a garbled request thanks to >> exactly-once receive semantics >> - we don't have a problem with the timing of registration and >> invalidation on the initiator because the PD's DMA lkey is used >> - we do have a problem with DMA unmap >> Using only persistently registered and DMA mapped Send buffers >> should avoid the need to signal all Sends. However, when page >> cache pages are involved, then the Send needs to be signaled, >> and the pages unmapped only after Send completion, to be completely >> safe. > > How do you know when that happens? The RPC Call send path sets up the Send SGE array. If it includes page cache pages, it can set IB_SEND_SIGNALED. The SGE array and the ib_cqe for the send are in the same data structure, so the Send completion handler can find the SGE array and figure out what needs to be unmapped. The only problem is if a POSIX signal fires. In that case the data structure can be released before the Send completion fires, and we get touch-after-free in the completion handler. I'm thinking that it just isn't going to be practical to handle unmapping this way, and I should just revert back to using RDMA Read instead of adding page cache pages to the Send SGE. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
