On 10/26/2018 05:52 PM, Jason Gunthorpe wrote:
On Fri, Oct 26, 2018 at 04:55:29PM +0200, Jörn Schumacher wrote:
Hi all,
I am trying to register an mmap'ed memory region as send buffer using
ibv_reg_mr (via a libfabric call).
The address is a virtual address that has been mmap'ed from a kernel address
in a custom kernel driver. The mapping uses remap_pfn_range:
vma->vm_flags |= VM_DONTEXPAND;
vma->vm_flags |= VM_DONTDUMP;
vma->vm_flags |= VM_LOCKED;
remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, size,
vma->vm_page_prot)
The ibv_reg_mr call fails with -22 (Invalid argument). The call succeeds if
I replace the buffer with a simple malloc'ed buffer.
Why would this not work with mmap'ed memory? Is there a way of mmap'ing the
kernel address to user space that would allow the memory registration?
mmap'd memory has to be comaptible with get_user_pages and
remap_pfn_range does not produce memory like that.
Thanks Jason. CC'ing Markus who wrote the driver I am talking about.
This driver maintains a large, contiguous chunk of non-fragmented
memory. We know already the physical address of this buffer and it is
pinned, so the call to get_user_pages seems redundant.
Is there a way to go around ibv_reg_mr and still use addresses from this
memory chunk in ibv_post_send/ibv_post_recv calls?
Jörn
