On 11/19/2018 10:09 PM, Jason Gunthorpe wrote:
On Mon, Nov 19, 2018 at 09:45:24PM -0500, Tom Talpey wrote:
On 11/19/2018 5:46 PM, Jason Gunthorpe wrote:
On Mon, Nov 19, 2018 at 09:42:56PM +0000, Mora, Jorge wrote:
Hello Chuck,
I am confused, is it the whole purpose of RDMA is to place the data
directly into the memory location given by the virtual address or
offset? What you are saying is that this offset is not the actual
memory address and so the driver must map this offset to the actual
address?
All RDMA on the wire addresses are 'virtual' and always go through
translation hardware to reach a physical memory address. Even if you
have a virtual address that matches your physical address there will
still be translation hardware to enforce security.
The direct placement refers to not copying data, not to using physical
memory addresses throughout the protocol design.
So it is better to use a virtual address that doesn't disclose any
information about the host's configuration to the network, such as
physical locations of kernel memory.
Agreed. And, this is a very subtle but very powerful aspect of FRWR
registration, that the high-order bits of the "virtual address" can
basically be any value of the requestor's (client's) choice.
Note, the low-order bits, up to the maximum offset used by the upper
layer (e.g. a few MB for NFS), must be reserved in order to allow RDMA
segments to iterate over the range of bytes in the region. That's only
24 or so bits out of the available 64, here however.
Why does it depend on protocol?
Because the maximum payload being RDMA's is an upper layer matter. The
idea here is to provide a certain number of "invariant" bits in the
offset / virtual address. Chuck proposes reserving just 32 bits, leaving
32 bits of range, quite comfy.
So long as the adaptor can compute VA + Offset = Physical the upper
bits can be changed.
Absolutely, but if the offset walks into the high-order invariant bits,
they won't be invariant any longer. It would still work, but it would
defeat the wireshark trick that Chuck wants to play.
For many adaptors that means PAGE_SIZE or huge
page size level alignment..
I'm not sure I understand this. The FRWR hides the native page size, by
providing the page frames in the local work request. Their size isn't
known to the protocol.
Tom.
