Hi, Shachar Raindel wrote on Fri, Apr 03, 2015 at 11:49:13AM +0000: > > couldn't get it to work - ibv_reg_mr would return EINVAL on an address > > obtained by mmap. > > Were you mmaping a normal disk file, or was the mmap targeting an MMIO of > another hardware device? mmap of a normal disk file should work also with > normal memory registration, assuming you are providing the proper length. On a proper file. It was a year or two ago, I actually tried again now and it does seem to work alright even on older kernels... I wonder what I did wrong back then! > mmap of the MMIO area of another hardware device (i.e. interfacing an FPGA, > NVRAM, or similar things) requires some code changes on both sides. The > current kernel code in the infiniband side is using get_user_pages, which > does not support MMIO pages. The proposed PeerDirect patches [1] allows peer > device to declare ownership of virtual address ranges, and enable such > registration. However, these patches are have not yet been merged upstream. Interesting, I don't need this right now but it's good to know. > > Conceptually as well I'm not sure how it's supposed to work, mmap should > > only actually issue reads when memory access issues page faults (give or > > take suitable readahead logic), but I don't think direct memory access > > from the IB/RDMA adapter would issue such page faults ? > > You are correct. RDMA adapters without ODP support do not issue page faults. > Instead, during memory registration, the ib_umem code calls get_user_pages, > which ensures all relevant pages are in memory, and pins them as needed. > > > Likewise on writes, would need the kernel to notice memory has been > > written and pages are dirty/needs flushing. > > > > Similarly, when deregistering a writable memory region, the kernel driver > marks all pages as dirty before unpinning them. You can see the code doing > this in [2]. Ok this makes sense, thanks for clearing it up. > Liran Liss gave a presentation about ODP at OFA [3]. The technology is > available for ConnectIB devices using the most recent firmware and kernel > versions above 3.19. I don't have any recent kernel around, but I'll give it a shot next week. (I'm working on a userspace file server, nfs-ganesha, so being able to mmap a file and use it directly for I/O is very promising for me!) > [1] http://www.spinics.net/lists/linux-rdma/msg21770.html > [2] http://lxr.free-electrons.com/source/drivers/infiniband/core/umem.c#L62 > [3] https://www.openfabrics.org/images/Workshops_2014/DevWorkshop/presos/Tuesday/pdf/09.30_2014_OFA_Workshop_ODP_update_final.pdf and https://www.youtube.com/watch?v=KbrlsXQbHCw Thanks for the detailed reply, I'll dig a bit further and come back straight to the list if need to. -- Dominique Martinet -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
