On 11/19/2018 5:46 PM, Jason Gunthorpe wrote:
On Mon, Nov 19, 2018 at 09:42:56PM +0000, Mora, Jorge wrote:
Hello Chuck,
I am confused, is it the whole purpose of RDMA is to place the data
directly into the memory location given by the virtual address or
offset? What you are saying is that this offset is not the actual
memory address and so the driver must map this offset to the actual
address?
All RDMA on the wire addresses are 'virtual' and always go through
translation hardware to reach a physical memory address. Even if you
have a virtual address that matches your physical address there will
still be translation hardware to enforce security.
The direct placement refers to not copying data, not to using physical
memory addresses throughout the protocol design.
So it is better to use a virtual address that doesn't disclose any
information about the host's configuration to the network, such as
physical locations of kernel memory.
Agreed. And, this is a very subtle but very powerful aspect of FRWR
registration, that the high-order bits of the "virtual address" can
basically be any value of the requestor's (client's) choice.
Note, the low-order bits, up to the maximum offset used by the upper
layer (e.g. a few MB for NFS), must be reserved in order to allow RDMA
segments to iterate over the range of bytes in the region. That's only
24 or so bits out of the available 64, here however.
For other registration methods, this is not always the case. It is
definitely best to leverage this important property for FRWR.
SMB Direct in Windows does a similar swizzling, by the way. We have
taken several different approaches to the format, over time.
Tom.
