> On Sep 6, 2017, at 10:29 AM, Sagi Grimberg <sagi@xxxxxxxxxxx> wrote: > > >>> Question, what happens in direct-io for example? Can a mapped buffer be >>> reclaimed/free'd before the send completion arrives? >> Good Q! RPC completion allows memory containing the arguments and >> results to be re-used. IIRC our conclusion was that a retransmitted >> Send could expose the wrong argument data on the wire in this case. >> Buffer re-use implies that the RPC has completed. Either a matching >> RPC Reply was received, or the RPC was terminated via a POSIX signal. >> If the client has already received an RPC Reply for this transaction, >> a previous transmission of the RPC Call has already executed on the >> server, and this retransmission will be ignored. It's only purpose is >> to generate an appropriate RDMA ACK. >> A re-used buffer might be subsequently used for data that is sensitive, >> and the retransmission will expose that data on the wire. > > That was where I was going with this... > >> To protect >> against that, RPC can use a GSS flavor that protects confidentiality >> of RPC arguments and results. This would also require RPC-over-RDMA >> to use only RDMA Read to convey RPC Call messages. Send would be used >> only to convey the chunk lists, never data. >> Note that the buffers used to construct RPC Calls are always mapped >> and Send uses the local DMA key to post them. These can also be >> re-used immediately after RPC completion. The exposure risk there is >> of RPC headers and non-data arguments. > > I see, but how can the user know that that it needs to use RPCSEC_GSS > otherwise nfs/rdma might compromise sensitive data? And is this > a valid constraint? (just asking, you're the expert) sec=krb5p is used in cases where data on the wire must remain confidential. Otherwise, sensitive or no, data on the wire goes in the clear. But an administrator might not expect that other sensitive data on the client (not involved with NFS) can be placed on the wire by the vagaries of memory allocation and hardware retransmission, as exceptionally rare as that might be. Memory in which Send data resides is donated to the device until the Send completion fires: the ULP has no way to get it back in the meantime. ULPs can invalidate memory used for RDMA Read at any time, but Send memory is registered with the local DMA key (as anything else is just as expensive as an RDMA data transfer). The immediate solution is to never use Send to move file data directly. It will always have to be copied into a buffer or we use RDMA Read. These buffers contain only data that is destined for the wire. Does that close the unwanted exposure completely? If the HCA can guarantee that all Sends complete quickly (either successful, flush, or time out after a few seconds) then it could be fair to make RPC completion also wait for Send completion. Otherwise, a ^C on a file operation targeting an unreachable server will hang indefinitely. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
