> On Sep 6, 2017, at 7:54 AM, Sagi Grimberg <sagi@xxxxxxxxxxx> wrote: > >> Hi Jason, Sagi- > > Hey Chuck, > >> As we discussed a few weeks ago, this patch series implements the >> following: >> - Send SGEs are now managed via lock-less, wait-free circular queues >> - Send SGEs referring to page cache pages are DMA unmapped during >> Send completion >> - Send completions are batched to reduce interrupts, but still >> provide a periodic heartbeat signal for SQ housekeeping >> - The circular queue prevents Send Queue overflow >> The purpose of this change is to address the issue Sagi reported >> where the HCA continues to retry a delayed Send request _after_ RPC >> completion, resulting in a DMA error. > > Question, what happens in direct-io for example? Can a mapped buffer be > reclaimed/free'd before the send completion arrives? Good Q! RPC completion allows memory containing the arguments and results to be re-used. IIRC our conclusion was that a retransmitted Send could expose the wrong argument data on the wire in this case. Buffer re-use implies that the RPC has completed. Either a matching RPC Reply was received, or the RPC was terminated via a POSIX signal. If the client has already received an RPC Reply for this transaction, a previous transmission of the RPC Call has already executed on the server, and this retransmission will be ignored. It's only purpose is to generate an appropriate RDMA ACK. A re-used buffer might be subsequently used for data that is sensitive, and the retransmission will expose that data on the wire. To protect against that, RPC can use a GSS flavor that protects confidentiality of RPC arguments and results. This would also require RPC-over-RDMA to use only RDMA Read to convey RPC Call messages. Send would be used only to convey the chunk lists, never data. Note that the buffers used to construct RPC Calls are always mapped and Send uses the local DMA key to post them. These can also be re-used immediately after RPC completion. The exposure risk there is of RPC headers and non-data arguments. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
