On Wed, Oct 31, 2018 at 03:59:59PM +0100, Jörn Schumacher wrote: > > > On 10/26/2018 05:52 PM, Jason Gunthorpe wrote: > > On Fri, Oct 26, 2018 at 04:55:29PM +0200, Jörn Schumacher wrote: > > > Hi all, > > > > > > I am trying to register an mmap'ed memory region as send buffer using > > > ibv_reg_mr (via a libfabric call). > > > > > > The address is a virtual address that has been mmap'ed from a kernel address > > > in a custom kernel driver. The mapping uses remap_pfn_range: > > > > > > > > > vma->vm_flags |= VM_DONTEXPAND; > > > vma->vm_flags |= VM_DONTDUMP; > > > vma->vm_flags |= VM_LOCKED; > > > remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, size, > > > vma->vm_page_prot) > > > > > > > > > The ibv_reg_mr call fails with -22 (Invalid argument). The call succeeds if > > > I replace the buffer with a simple malloc'ed buffer. > > > > > > Why would this not work with mmap'ed memory? Is there a way of mmap'ing the > > > kernel address to user space that would allow the memory registration? > > > > mmap'd memory has to be comaptible with get_user_pages and > > remap_pfn_range does not produce memory like that. > > > Thanks Jason. CC'ing Markus who wrote the driver I am talking about. > > This driver maintains a large, contiguous chunk of non-fragmented memory. We > know already the physical address of this buffer and it is pinned, so the > call to get_user_pages seems redundant. > > Is there a way to go around ibv_reg_mr and still use addresses from this > memory chunk in ibv_post_send/ibv_post_recv calls? AFAIK this other driver would have to somehow provide a system call to create the IB MR via the kernel API that can use physical addresses. Better would be to arrange so the other driver creates a user VMA that is backed by struct pages so get_user_pages works right Jason
