> From: Christoph Lameter [mailto:cl@xxxxxxxxx] > Sent: Tuesday, June 06, 2017 6:30 PM Hi Chistoph, > The OS needs to provide the protection if that is the case and then it probably is > not a HPC device. We are talking about RDMA high end devices here. The design > here is for performance and low latency. I dont know of any devices in use in > HPC or in HFT that have these tiny memory sizes. > Mostly these devices are already engineeded for mmapping. > > Is this for some kind of embedded device? > In our case, this is the HCA. Device memory could be a scarce resource, and even allocating a 4K page per process could wasteful in some systems. An allocation API allows allocations of any size. > > Note that the device memory does not necessary have to be mapped to > > the CPU= . > > i.e. is not necessary accessible by PCI, and can only be accessed by RDMA. > > This is why we can't use MMAP for all cases and a dedicated > > allocation and copy functions are needed. > > Can we come up with some sort of ioctl API then to write to the devices > inaccessible memory? There must be other drives outside of the RDMA tree that > have similar requirements and that may already have implemented some > version of it. > > This seems to be a very specialized application that may be device specific. ioctls > are usually used then. The host to dev memcopy is often a user-space only action, so ioctl() doesn't work for us. The memcpy API allows efficient data transfer regardless of the implementation. I also don't think that always mandating a direct mmap() is the solution. First, not in all cases such a mapping exists. Some devices have only limited BAR space, which can only be dynamically mapped to a small window at a time. In this case, you might need to move the window before the access. Other device might not be able to map device memory to the CPU at all. Even if a direct mapping is possible, you don't expose mmap()'d IO memory directly to the application. There are details such as optimizing for write-combining, locking, and flushing that the application shouldn't be aware of. --Liran -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
