On Mon, Nov 19, 2018 at 09:45:24PM -0500, Tom Talpey wrote: > On 11/19/2018 5:46 PM, Jason Gunthorpe wrote: > > On Mon, Nov 19, 2018 at 09:42:56PM +0000, Mora, Jorge wrote: > > > Hello Chuck, > > > > > > I am confused, is it the whole purpose of RDMA is to place the data > > > directly into the memory location given by the virtual address or > > > offset? What you are saying is that this offset is not the actual > > > memory address and so the driver must map this offset to the actual > > > address? > > > > All RDMA on the wire addresses are 'virtual' and always go through > > translation hardware to reach a physical memory address. Even if you > > have a virtual address that matches your physical address there will > > still be translation hardware to enforce security. > > > > The direct placement refers to not copying data, not to using physical > > memory addresses throughout the protocol design. > > > > So it is better to use a virtual address that doesn't disclose any > > information about the host's configuration to the network, such as > > physical locations of kernel memory. > > Agreed. And, this is a very subtle but very powerful aspect of FRWR > registration, that the high-order bits of the "virtual address" can > basically be any value of the requestor's (client's) choice. > > Note, the low-order bits, up to the maximum offset used by the upper > layer (e.g. a few MB for NFS), must be reserved in order to allow RDMA > segments to iterate over the range of bytes in the region. That's only > 24 or so bits out of the available 64, here however. Why does it depend on protocol? So long as the adaptor can compute VA + Offset = Physical the upper bits can be changed. For many adaptors that means PAGE_SIZE or huge page size level alignment.. Jason
