Hi Dan, sorry for the late reply. This sat in my outbox for a while almost fully written. On Wed, Oct 27, 2021 at 09:23:42AM +0100, Dan Greenfield wrote: > Quick Question: would I be correct in assuming that if I mmap()ed each 8-64MB (chunk) file from XFS, and then did RDMA from the mmap region, that it would first be copied from NVMe into DRAM (does this bypass CPU?) and *then* be copied across RDMA, rather than directly be copied from NVMe by RDMA? Or does O_DIRECT properly allow bypass straight to NVMe for RDMA? The answer is: it depends. mmap on a non-DAX file system always copied into DRAM. mmap on a DAX file system (that is using pmem) can map the "storage" directly into memory, in which case some RDMA setups can DMA without a copy. But with a plain old SSD there is no path available to userspace to transfer without copying to DRAM. If OTOH you use in-kernel NVMe over fabics target, it can directly transfers from the SSDs in some circumstances. Currently that does require using the SSD directly without a file system, but with a little more work it could also work using a file system.
