On 10/31/2018 05:20 PM, Jason Gunthorpe wrote:
On Wed, Oct 31, 2018 at 03:59:59PM +0100, Jörn Schumacher wrote:
On 10/26/2018 05:52 PM, Jason Gunthorpe wrote:
On Fri, Oct 26, 2018 at 04:55:29PM +0200, Jörn Schumacher wrote:
Hi all,
I am trying to register an mmap'ed memory region as send buffer using
ibv_reg_mr (via a libfabric call).
The address is a virtual address that has been mmap'ed from a kernel address
in a custom kernel driver. The mapping uses remap_pfn_range:
vma->vm_flags |= VM_DONTEXPAND;
vma->vm_flags |= VM_DONTDUMP;
vma->vm_flags |= VM_LOCKED;
remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, size,
vma->vm_page_prot)
The ibv_reg_mr call fails with -22 (Invalid argument). The call succeeds if
I replace the buffer with a simple malloc'ed buffer.
Why would this not work with mmap'ed memory? Is there a way of mmap'ing the
kernel address to user space that would allow the memory registration?
mmap'd memory has to be comaptible with get_user_pages and
remap_pfn_range does not produce memory like that.
Thanks Jason. CC'ing Markus who wrote the driver I am talking about.
This driver maintains a large, contiguous chunk of non-fragmented memory. We
know already the physical address of this buffer and it is pinned, so the
call to get_user_pages seems redundant.
Is there a way to go around ibv_reg_mr and still use addresses from this
memory chunk in ibv_post_send/ibv_post_recv calls?
AFAIK this other driver would have to somehow provide a system call to
create the IB MR via the kernel API that can use physical addresses.
Better would be to arrange so the other driver creates a user VMA that
is backed by struct pages so get_user_pages works right
Do you know an alternative to remap_pfn_range that would do the memory
mapping *and* be compatible with get_user_pages?
Let me know if you think there is a more appropriate mailing list for
this, as this is not purely RDMA-related anymore.
Jörn
