On Thu, May 30, 2019 at 09:58:53PM +0000, Adit Ranadive wrote:
> On 5/30/19 7:34 AM, Yuval Shaia wrote:
> > On Thu, May 30, 2019 at 12:37:18PM +0000, Michal Kalderon wrote:
> >>> From: linux-rdma-owner@xxxxxxxxxxxxxxx <linux-rdma-
> >>> owner@xxxxxxxxxxxxxxx> On Behalf Of Yuval Shaia
> >>>
> >>> The virtual address that is registered is used as a base for any address passed
> >>> later in post_recv and post_send operations.
> >>>
> >>> On a virtualized environment this is not correct.
>
> Sorry but I think that really depends on the virtualized environment..
You are right.
I'll change it to something like this:
On a virtualized environment with emulated device this is not correct.
(in case you folks from VMWare solve it in some other way then i'll
s/a/some)
>
> >>>
> >>> A guest cannot register its memory so hypervisor maps the guest physical
> >>> address to a host virtual address and register it with the HW. Later on, at
> >>> datapath phase, the guest fills the SGEs with addresses from its address
> >>> space.
> >>> Since HW cannot access guest virtual address space an extra translation is
> >>> needed to map those addresses to be based on the host virtual address that
> >>> was registered with the HW.
> >>> This datapath interference affects performances.
>
> Doesn't this also mean that you cannot do remote reads/writes between guests or
> is there an additional mechanism that exchanges translations as well?
Well, haven't yet evaluated the affect on rdma read/write ops but looks
like it should help here either.
>
> >>>
> >>> To avoid this, a logical separation between the address that is registered and
> >>> the address that is used as a offset at datapath phase is needed.
> >>> This separation is already implemented in the lower layer part
> >>> (ibv_cmd_reg_mr) but blocked at the API level.
> >>>
> >>> Fix it by introducing a new API function which accepts an address from guest
> >>> virtual address space as well, to be used as offset for later datapath
> >>> operations.
> >>>
> >> Could you give an example of how an app would use this new API? How will
> >> It receive the new hca_va addresss ?
> >
> > In my use case an application is device emulation that runs in the context
> > of a userspace process in the host.
> > This (virtual) device receives from guest driver a dma address (in form of
> > scatter-gather list) along with guest user-space virtual address. This is
> > done i guess by any device driver. The dma-list is mapped to host virtual
> > address that is used as the addr argument and the guest-user-space virtual
> > address is used as the iova argument.
> > From this moment the guest userspace can continue using the virtual address
> > as a base for any further post_send and post_recv operations, no extra
> > translation is needed.
>
> That's interesting. So the guest would still do a ibv_reg_mr and the device
> would re-send the mappings using ibv_reg_mr_iova?
Exactly!
> How would you prevent applications from using/misusing ibv_reg_mr_iova by
> passing arbitrary values to hca_va or is that even a problem? What if the
Well, as long as the addresses in the post_send/recv related to the iova i
don't see any problem with using it.
For example Jason utilize this function for zero based MR where iova is 0
and the addresses in the post_send/recv are offset from zero.
> underlying device doesn't support zero-based MRs? I guess that should be
> caught in the driver..
Well, this patch covers the current providers and i didn't find one that is
not supporting it.
>
> > Another use case is as Jason just mention - for the zero based MR.
> >
> >> And which virtualized environment are you referring to? Para-virtualized ?
> >> (since you don't have this problem with sriov - right ? )
> >
> > Yes, para-virt. sriov doesn't need it since it is driven by a vendor driver
> > in guest that interact directly with the HW via iommu that translates the
> > addresses, right?
> >
> >> Thanks,
> >> Michal
> >>
> >>> Signed-off-by: Yuval Shaia <yuval.shaia@xxxxxxxxxx>
> >>> ---
> >>> v0 -> v1:
> >>> * Change reg_mr callback signature instead of adding new callback
> >>> * Add the new API to libibverbs/libibverbs.map.in
> >>> v1 -> v2:
> >>> * Do not modify reg_mr signature for version 1.0
> >>> * Add note to man page
> >>> v2 -> v3:
> >>> * Rename function to reg_mr_iova (and arg-name to iova)
> >>> * Some checkpatch issues not related to this fix but detected now
> >>> * s/__FUNCTION__/__func
> >>> * WARNING: function definition argument 'void *' should
> >>> also have an identifier name
> >>> ---
> >>> libibverbs/driver.h | 2 +-
> >>> libibverbs/dummy_ops.c | 2 +-
> >>> libibverbs/libibverbs.map.in | 1 +
> >>> libibverbs/man/ibv_reg_mr.3 | 15 +++++++++++++--
> >>> libibverbs/verbs.c | 23 ++++++++++++++++++++++-
> >>> libibverbs/verbs.h | 7 +++++++
> >>> providers/bnxt_re/verbs.c | 6 +++---
> >>> providers/bnxt_re/verbs.h | 2 +-
> >>> providers/cxgb3/iwch.h | 4 ++--
> >>> providers/cxgb3/verbs.c | 9 +++++----
> >>> providers/cxgb4/libcxgb4.h | 4 ++--
> >>> providers/cxgb4/verbs.c | 9 +++++----
> >>> providers/hfi1verbs/hfiverbs.h | 4 ++--
> >>> providers/hfi1verbs/verbs.c | 8 ++++----
> >>> providers/hns/hns_roce_u.h | 2 +-
> >>> providers/hns/hns_roce_u_verbs.c | 6 +++---
> >>> providers/i40iw/i40iw_umain.h | 3 ++-
> >>> providers/i40iw/i40iw_uverbs.c | 8 ++++----
> >>> providers/ipathverbs/ipathverbs.h | 4 ++--
> >>> providers/ipathverbs/verbs.c | 8 ++++----
> >>> providers/mlx4/mlx4.h | 4 ++--
> >>> providers/mlx4/verbs.c | 7 +++----
> >>> providers/mlx5/mlx5.h | 4 ++--
> >>> providers/mlx5/verbs.c | 7 +++----
> >>> providers/mthca/ah.c | 3 ++-
> >>> providers/mthca/mthca.h | 4 ++--
> >>> providers/mthca/verbs.c | 6 +++---
> >>> providers/nes/nes_umain.h | 3 ++-
> >>> providers/nes/nes_uverbs.c | 9 ++++-----
> >>> providers/ocrdma/ocrdma_main.h | 4 ++--
> >>> providers/ocrdma/ocrdma_verbs.c | 10 ++++------
> >>> providers/qedr/qelr_main.h | 4 ++--
> >>> providers/qedr/qelr_verbs.c | 11 ++++-------
> >>> providers/qedr/qelr_verbs.h | 4 ++--
> >>> providers/rxe/rxe.c | 6 +++---
> >>> providers/vmw_pvrdma/pvrdma.h | 4 ++--
> >>> providers/vmw_pvrdma/verbs.c | 7 +++----
> >>> 37 files changed, 130 insertions(+), 94 deletions(-)
> >>>
> >>> diff --git a/libibverbs/driver.h b/libibverbs/driver.h index e4d624b2..ef27259a
> >>> 100644
> >>> --- a/libibverbs/driver.h
> >>> +++ b/libibverbs/driver.h
> >>> @@ -338,7 +338,7 @@ struct verbs_context_ops {
> >>> uint64_t dm_offset, size_t length,
> >>> unsigned int access);
> >>> struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> - int access);
> >>> + uint64_t hca_va, int access);
> >>> int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
> >>> int (*rereg_mr)(struct verbs_mr *vmr, int flags, struct ibv_pd *pd,
> >>> void *addr, size_t length, int access); diff --git
> >>> a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c index
> >>> c861c3a0..61a8fbdf 100644
> >>> --- a/libibverbs/dummy_ops.c
> >>> +++ b/libibverbs/dummy_ops.c
> >>> @@ -410,7 +410,7 @@ static struct ibv_mr *reg_dm_mr(struct ibv_pd *pd,
> >>> struct ibv_dm *dm, }
> >>>
> >>> static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> - int access)
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> errno = ENOSYS;
> >>> return NULL;
> >>> diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in index
> >>> 87a1b9fc..aa0425ba 100644
> >>> --- a/libibverbs/libibverbs.map.in
> >>> +++ b/libibverbs/libibverbs.map.in
> >>> @@ -94,6 +94,7 @@ IBVERBS_1.1 {
> >>> ibv_query_srq;
> >>> ibv_rate_to_mbps;
> >>> ibv_reg_mr;
> >>> + ibv_reg_mr_iova;
> >>> ibv_register_driver;
> >>> ibv_rereg_mr;
> >>> ibv_resize_cq;
> >>> diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
> >>> index 631e5fe8..6c4c1434 100644
> >>> --- a/libibverbs/man/ibv_reg_mr.3
> >>> +++ b/libibverbs/man/ibv_reg_mr.3
> >>> @@ -3,7 +3,7 @@
> >>> .\"
> >>> .TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
> >>> .SH "NAME"
> >>> -ibv_reg_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
> >>> +ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a
> >>> +memory region (MR)
> >>> .SH "SYNOPSIS"
> >>> .nf
> >>> .B #include <infiniband/verbs.h>
> >>> @@ -11,6 +11,10 @@ ibv_reg_mr, ibv_dereg_mr \- register or deregister a
> >>> memory region (MR) .BI "struct ibv_mr *ibv_reg_mr(struct ibv_pd " "*pd" ",
> >>> void " "*addr" ,
> >>> .BI " size_t " "length" ", int " "access" );
> >>> .sp
> >>> +.BI "struct ibv_mr *ibv_reg_mr_iova(struct ibv_pd " "*pd" ", void " "*addr"
> >>> ,
> >>> +.BI " size_t " "length" ", uint64_t " "hca_va" ,
> >>> +.BI " int " "access" );
> >>> +.sp
> >>> .BI "int ibv_dereg_mr(struct ibv_mr " "*mr" ); .fi .SH "DESCRIPTION"
> >>> @@ -52,11 +56,18 @@ Local read access is always enabled for the MR.
> >>> .PP
> >>> To create an implicit ODP MR, IBV_ACCESS_ON_DEMAND should be set,
> >>> addr should be 0 and length should be SIZE_MAX.
> >>> .PP
> >>> +.B ibv_reg_mr_iova()
> >>> +Special variant of memory registration used when addresses passed to
> >>> +ibv_post_send and ibv_post_recv are relative to base address from a
> >>> +different address space than .I addr\fR. The argument .I hca_va\fR is
> >>> +the new base address.
> >>> +.PP
> >>> .B ibv_dereg_mr()
> >>> deregisters the MR
> >>> .I mr\fR.
> >>> .SH "RETURN VALUE"
> >>> -.B ibv_reg_mr()
> >>> +.B ibv_reg_mr() / ibv_reg_mr_iova()
> >>> returns a pointer to the registered MR, or NULL if the request fails.
> >>> The local key (\fBL_Key\fR) field
> >>> .B lkey
> >>> diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c index 1766b9f5..56502ea5
> >>> 100644
> >>> --- a/libibverbs/verbs.c
> >>> +++ b/libibverbs/verbs.c
> >>> @@ -312,7 +312,28 @@ LATEST_SYMVER_FUNC(ibv_reg_mr, 1_1,
> >>> "IBVERBS_1.1",
> >>> if (ibv_dontfork_range(addr, length))
> >>> return NULL;
> >>>
> >>> - mr = get_ops(pd->context)->reg_mr(pd, addr, length, access);
> >>> + mr = get_ops(pd->context)->reg_mr(pd, addr, length, (uint64_t)
> >>> addr,
> >>> + access);
> >>> + if (mr) {
> >>> + mr->context = pd->context;
> >>> + mr->pd = pd;
> >>> + mr->addr = addr;
> >>> + mr->length = length;
> >>> + } else
> >>> + ibv_dofork_range(addr, length);
> >>> +
> >>> + return mr;
> >>> +}
> >>> +
> >>> +struct ibv_mr *ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> + uint64_t iova, int access)
> >>> +{
> >>> + struct ibv_mr *mr;
> >>> +
> >>> + if (ibv_dontfork_range(addr, length))
> >>> + return NULL;
> >>> +
> >>> + mr = get_ops(pd->context)->reg_mr(pd, addr, length, iova, access);
> >>> if (mr) {
> >>> mr->context = pd->context;
> >>> mr->pd = pd;
> >>> diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index cb2d8439..c205a24c
> >>> 100644
> >>> --- a/libibverbs/verbs.h
> >>> +++ b/libibverbs/verbs.h
> >>> @@ -2372,6 +2372,13 @@ static inline int ibv_close_xrcd(struct ibv_xrcd
> >>> *xrcd) struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr,
> >>> size_t length, int access);
> >>>
> >>> +/**
> >>> + * ibv_reg_mr_iova - Register a memory region with a virtual offset
> >>> + * address
> >>> + */
> >>> +struct ibv_mr *ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> + uint64_t iova, int access);
> >>> +
> >>>
> >>> enum ibv_rereg_mr_err_code {
> >>> /* Old MR is valid, invalid input */
> >>> diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index
> >>> bec382b3..f85c277b 100644
> >>> --- a/providers/bnxt_re/verbs.c
> >>> +++ b/providers/bnxt_re/verbs.c
> >>> @@ -132,7 +132,7 @@ int bnxt_re_free_pd(struct ibv_pd *ibvpd) }
> >>>
> >>> struct ibv_mr *bnxt_re_reg_mr(struct ibv_pd *ibvpd, void *sva, size_t len,
> >>> - int access)
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct bnxt_re_mr *mr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -142,8 +142,8 @@ struct ibv_mr *bnxt_re_reg_mr(struct ibv_pd *ibvpd,
> >>> void *sva, size_t len,
> >>> if (!mr)
> >>> return NULL;
> >>>
> >>> - if (ibv_cmd_reg_mr(ibvpd, sva, len, (uintptr_t)sva, access, &mr-
> >>>> vmr,
> >>> - &cmd, sizeof(cmd), &resp.ibv_resp, sizeof(resp))) {
> >>> + if (ibv_cmd_reg_mr(ibvpd, sva, len, hca_va, access, &mr->vmr,
> >>> &cmd,
> >>> + sizeof(cmd), &resp.ibv_resp, sizeof(resp))) {
> >>> free(mr);
> >>> return NULL;
> >>> }
> >>> diff --git a/providers/bnxt_re/verbs.h b/providers/bnxt_re/verbs.h index
> >>> 6e5f092b..af5f35f9 100644
> >>> --- a/providers/bnxt_re/verbs.h
> >>> +++ b/providers/bnxt_re/verbs.h
> >>> @@ -62,7 +62,7 @@ int bnxt_re_query_port(struct ibv_context *uctx,
> >>> uint8_t port, struct ibv_pd *bnxt_re_alloc_pd(struct ibv_context *uctx); int
> >>> bnxt_re_free_pd(struct ibv_pd *ibvpd); struct ibv_mr
> >>> *bnxt_re_reg_mr(struct ibv_pd *ibvpd, void *buf, size_t len,
> >>> - int ibv_access_flags);
> >>> + uint64_t hca_va, int ibv_access_flags);
> >>> int bnxt_re_dereg_mr(struct verbs_mr *vmr);
> >>>
> >>> struct ibv_cq *bnxt_re_create_cq(struct ibv_context *uctx, int ncqe, diff --
> >>> git a/providers/cxgb3/iwch.h b/providers/cxgb3/iwch.h index
> >>> a6fea23d..ccb11729 100644
> >>> --- a/providers/cxgb3/iwch.h
> >>> +++ b/providers/cxgb3/iwch.h
> >>> @@ -150,8 +150,8 @@ extern int iwch_query_port(struct ibv_context
> >>> *context, uint8_t port, extern struct ibv_pd *iwch_alloc_pd(struct
> >>> ibv_context *context); extern int iwch_free_pd(struct ibv_pd *pd);
> >>>
> >>> -extern struct ibv_mr *iwch_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +extern struct ibv_mr *iwch_reg_mr(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> + uint64_t hca_va, int access);
> >>> extern int iwch_dereg_mr(struct verbs_mr *mr);
> >>>
> >>> struct ibv_cq *iwch_create_cq(struct ibv_context *context, int cqe, diff --git
> >>> a/providers/cxgb3/verbs.c b/providers/cxgb3/verbs.c index
> >>> 67eb588b..091d7a67 100644
> >>> --- a/providers/cxgb3/verbs.c
> >>> +++ b/providers/cxgb3/verbs.c
> >>> @@ -140,11 +140,12 @@ static struct ibv_mr *__iwch_reg_mr(struct ibv_pd
> >>> *pd, void *addr,
> >>> return &mhp->vmr.ibv_mr;
> >>> }
> >>>
> >>> -struct ibv_mr *iwch_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access)
> >>> +struct ibv_mr *iwch_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> - PDBG("%s addr %p length %ld\n", __FUNCTION__, addr, length);
> >>> - return __iwch_reg_mr(pd, addr, length, (uintptr_t) addr, access);
> >>> + PDBG("%s addr %p length %ld hca_va %p\n", __func__, addr, length,
> >>> + hca_va);
> >>> + return __iwch_reg_mr(pd, addr, length, hca_va, access);
> >>> }
> >>>
> >>> int iwch_dereg_mr(struct verbs_mr *vmr) diff --git
> >>> a/providers/cxgb4/libcxgb4.h b/providers/cxgb4/libcxgb4.h index
> >>> 7626a5f1..1ee94e4c 100644
> >>> --- a/providers/cxgb4/libcxgb4.h
> >>> +++ b/providers/cxgb4/libcxgb4.h
> >>> @@ -198,8 +198,8 @@ int c4iw_query_port(struct ibv_context *context,
> >>> uint8_t port, struct ibv_pd *c4iw_alloc_pd(struct ibv_context *context); int
> >>> c4iw_free_pd(struct ibv_pd *pd);
> >>>
> >>> -struct ibv_mr *c4iw_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +struct ibv_mr *c4iw_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>> int c4iw_dereg_mr(struct verbs_mr *vmr);
> >>>
> >>> struct ibv_cq *c4iw_create_cq(struct ibv_context *context, int cqe, diff --git
> >>> a/providers/cxgb4/verbs.c b/providers/cxgb4/verbs.c index
> >>> f5c154e6..c8aeb8b4 100644
> >>> --- a/providers/cxgb4/verbs.c
> >>> +++ b/providers/cxgb4/verbs.c
> >>> @@ -142,11 +142,12 @@ static struct ibv_mr *__c4iw_reg_mr(struct ibv_pd
> >>> *pd, void *addr,
> >>> return &mhp->vmr.ibv_mr;
> >>> }
> >>>
> >>> -struct ibv_mr *c4iw_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access)
> >>> +struct ibv_mr *c4iw_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> - PDBG("%s addr %p length %ld\n", __func__, addr, length);
> >>> - return __c4iw_reg_mr(pd, addr, length, (uintptr_t) addr, access);
> >>> + PDBG("%s addr %p length %ld hca_va %p\n", __func__, addr, length,
> >>> + hca_va);
> >>> + return __c4iw_reg_mr(pd, addr, length, hca_va, access);
> >>> }
> >>>
> >>> int c4iw_dereg_mr(struct verbs_mr *vmr) diff --git
> >>> a/providers/hfi1verbs/hfiverbs.h b/providers/hfi1verbs/hfiverbs.h index
> >>> 070a01c9..b9e91d80 100644
> >>> --- a/providers/hfi1verbs/hfiverbs.h
> >>> +++ b/providers/hfi1verbs/hfiverbs.h
> >>> @@ -204,8 +204,8 @@ struct ibv_pd *hfi1_alloc_pd(struct ibv_context *pd);
> >>>
> >>> int hfi1_free_pd(struct ibv_pd *pd);
> >>>
> >>> -struct ibv_mr *hfi1_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +struct ibv_mr *hfi1_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>>
> >>> int hfi1_dereg_mr(struct verbs_mr *vmr);
> >>>
> >>> diff --git a/providers/hfi1verbs/verbs.c b/providers/hfi1verbs/verbs.c index
> >>> ff001f6d..275f8d51 100644
> >>> --- a/providers/hfi1verbs/verbs.c
> >>> +++ b/providers/hfi1verbs/verbs.c
> >>> @@ -129,8 +129,8 @@ int hfi1_free_pd(struct ibv_pd *pd)
> >>> return 0;
> >>> }
> >>>
> >>> -struct ibv_mr *hfi1_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access)
> >>> +struct ibv_mr *hfi1_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct verbs_mr *vmr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -141,8 +141,8 @@ struct ibv_mr *hfi1_reg_mr(struct ibv_pd *pd, void
> >>> *addr,
> >>> if (!vmr)
> >>> return NULL;
> >>>
> >>> - ret = ibv_cmd_reg_mr(pd, addr, length, (uintptr_t)addr, access, vmr,
> >>> - &cmd, sizeof cmd, &resp, sizeof resp);
> >>> + ret = ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr, &cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp));
> >>>
> >>> if (ret) {
> >>> free(vmr);
> >>> diff --git a/providers/hns/hns_roce_u.h b/providers/hns/hns_roce_u.h
> >>> index 65125e5a..2d35e00f 100644
> >>> --- a/providers/hns/hns_roce_u.h
> >>> +++ b/providers/hns/hns_roce_u.h
> >>> @@ -293,7 +293,7 @@ struct ibv_pd *hns_roce_u_alloc_pd(struct
> >>> ibv_context *context); int hns_roce_u_free_pd(struct ibv_pd *pd);
> >>>
> >>> struct ibv_mr *hns_roce_u_reg_mr(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> - int access);
> >>> + uint64_t hca_va, int access);
> >>> int hns_roce_u_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd,
> >>> void *addr, size_t length, int access); int
> >>> hns_roce_u_dereg_mr(struct verbs_mr *mr); diff --git
> >>> a/providers/hns/hns_roce_u_verbs.c b/providers/hns/hns_roce_u_verbs.c
> >>> index 9ca62c72..8cf21870 100644
> >>> --- a/providers/hns/hns_roce_u_verbs.c
> >>> +++ b/providers/hns/hns_roce_u_verbs.c
> >>> @@ -120,7 +120,7 @@ int hns_roce_u_free_pd(struct ibv_pd *pd) }
> >>>
> >>> struct ibv_mr *hns_roce_u_reg_mr(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> - int access)
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> int ret;
> >>> struct verbs_mr *vmr;
> >>> @@ -141,8 +141,8 @@ struct ibv_mr *hns_roce_u_reg_mr(struct ibv_pd
> >>> *pd, void *addr, size_t length,
> >>> if (!vmr)
> >>> return NULL;
> >>>
> >>> - ret = ibv_cmd_reg_mr(pd, addr, length, (uintptr_t)addr, access, vmr,
> >>> - &cmd, sizeof(cmd), &resp, sizeof(resp));
> >>> + ret = ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr, &cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp));
> >>> if (ret) {
> >>> free(vmr);
> >>> return NULL;
> >>> diff --git a/providers/i40iw/i40iw_umain.h b/providers/i40iw/i40iw_umain.h
> >>> index 80c1727c..2843c4bd 100644
> >>> --- a/providers/i40iw/i40iw_umain.h
> >>> +++ b/providers/i40iw/i40iw_umain.h
> >>> @@ -160,7 +160,8 @@ int i40iw_uquery_device(struct ibv_context *, struct
> >>> ibv_device_attr *); int i40iw_uquery_port(struct ibv_context *, uint8_t,
> >>> struct ibv_port_attr *); struct ibv_pd *i40iw_ualloc_pd(struct ibv_context
> >>> *); int i40iw_ufree_pd(struct ibv_pd *); -struct ibv_mr
> >>> *i40iw_ureg_mr(struct ibv_pd *, void *, size_t, int);
> >>> +struct ibv_mr *i40iw_ureg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>> int i40iw_udereg_mr(struct verbs_mr *vmr); struct ibv_cq
> >>> *i40iw_ucreate_cq(struct ibv_context *, int, struct ibv_comp_channel *, int);
> >>> int i40iw_uresize_cq(struct ibv_cq *, int); diff --git
> >>> a/providers/i40iw/i40iw_uverbs.c b/providers/i40iw/i40iw_uverbs.c index
> >>> 63b7206a..c054eb56 100644
> >>> --- a/providers/i40iw/i40iw_uverbs.c
> >>> +++ b/providers/i40iw/i40iw_uverbs.c
> >>> @@ -149,7 +149,8 @@ int i40iw_ufree_pd(struct ibv_pd *pd)
> >>> * @length: length of the memory
> >>> * @access: access allowed on this mr
> >>> */
> >>> -struct ibv_mr *i40iw_ureg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> int access)
> >>> +struct ibv_mr *i40iw_ureg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct verbs_mr *vmr;
> >>> struct i40iw_ureg_mr cmd;
> >>> @@ -161,9 +162,8 @@ struct ibv_mr *i40iw_ureg_mr(struct ibv_pd *pd, void
> >>> *addr, size_t length, int a
> >>>
> >>> cmd.reg_type = IW_MEMREG_TYPE_MEM;
> >>>
> >>> - if (ibv_cmd_reg_mr(pd, addr, length, (uintptr_t)addr,
> >>> - access, vmr, &cmd.ibv_cmd, sizeof(cmd),
> >>> - &resp, sizeof(resp))) {
> >>> + if (ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr,
> >>> &cmd.ibv_cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp))) {
> >>> fprintf(stderr, PFX "%s: Failed to register memory\n",
> >>> __func__);
> >>> free(vmr);
> >>> return NULL;
> >>> diff --git a/providers/ipathverbs/ipathverbs.h
> >>> b/providers/ipathverbs/ipathverbs.h
> >>> index cfb5cc38..694f1f44 100644
> >>> --- a/providers/ipathverbs/ipathverbs.h
> >>> +++ b/providers/ipathverbs/ipathverbs.h
> >>> @@ -183,8 +183,8 @@ struct ibv_pd *ipath_alloc_pd(struct ibv_context
> >>> *pd);
> >>>
> >>> int ipath_free_pd(struct ibv_pd *pd);
> >>>
> >>> -struct ibv_mr *ipath_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +struct ibv_mr *ipath_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>>
> >>> int ipath_dereg_mr(struct verbs_mr *vmr);
> >>>
> >>> diff --git a/providers/ipathverbs/verbs.c b/providers/ipathverbs/verbs.c
> >>> index de4722b2..505ea584 100644
> >>> --- a/providers/ipathverbs/verbs.c
> >>> +++ b/providers/ipathverbs/verbs.c
> >>> @@ -109,8 +109,8 @@ int ipath_free_pd(struct ibv_pd *pd)
> >>> return 0;
> >>> }
> >>>
> >>> -struct ibv_mr *ipath_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access)
> >>> +struct ibv_mr *ipath_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct verbs_mr *vmr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -121,8 +121,8 @@ struct ibv_mr *ipath_reg_mr(struct ibv_pd *pd, void
> >>> *addr,
> >>> if (!vmr)
> >>> return NULL;
> >>>
> >>> - ret = ibv_cmd_reg_mr(pd, addr, length, (uintptr_t)addr, access, vmr,
> >>> - &cmd, sizeof cmd, &resp, sizeof resp);
> >>> + ret = ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr, &cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp));
> >>> if (ret) {
> >>> free(vmr);
> >>> return NULL;
> >>> diff --git a/providers/mlx4/mlx4.h b/providers/mlx4/mlx4.h index
> >>> e9412c51..ae02221a 100644
> >>> --- a/providers/mlx4/mlx4.h
> >>> +++ b/providers/mlx4/mlx4.h
> >>> @@ -325,8 +325,8 @@ struct ibv_xrcd *mlx4_open_xrcd(struct ibv_context
> >>> *context,
> >>> struct ibv_xrcd_init_attr *attr);
> >>> int mlx4_close_xrcd(struct ibv_xrcd *xrcd);
> >>>
> >>> -struct ibv_mr *mlx4_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +struct ibv_mr *mlx4_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>> int mlx4_rereg_mr(struct verbs_mr *vmr, int flags, struct ibv_pd *pd,
> >>> void *addr, size_t length, int access); int
> >>> mlx4_dereg_mr(struct verbs_mr *vmr); diff --git a/providers/mlx4/verbs.c
> >>> b/providers/mlx4/verbs.c index eae75f1d..697ef769 100644
> >>> --- a/providers/mlx4/verbs.c
> >>> +++ b/providers/mlx4/verbs.c
> >>> @@ -275,7 +275,7 @@ int mlx4_close_xrcd(struct ibv_xrcd *ib_xrcd) }
> >>>
> >>> struct ibv_mr *mlx4_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> - int access)
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct verbs_mr *vmr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -286,9 +286,8 @@ struct ibv_mr *mlx4_reg_mr(struct ibv_pd *pd, void
> >>> *addr, size_t length,
> >>> if (!vmr)
> >>> return NULL;
> >>>
> >>> - ret = ibv_cmd_reg_mr(pd, addr, length, (uintptr_t) addr,
> >>> - access, vmr, &cmd, sizeof(cmd),
> >>> - &resp, sizeof(resp));
> >>> + ret = ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr, &cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp));
> >>> if (ret) {
> >>> free(vmr);
> >>> return NULL;
> >>> diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index
> >>> 3838e901..77b0b151 100644
> >>> --- a/providers/mlx5/mlx5.h
> >>> +++ b/providers/mlx5/mlx5.h
> >>> @@ -798,8 +798,8 @@ struct ibv_pd *mlx5_alloc_pd(struct ibv_context
> >>> *context); int mlx5_free_pd(struct ibv_pd *pd);
> >>>
> >>> struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd); -struct ibv_mr
> >>> *mlx5_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>> int mlx5_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd, void
> >>> *addr,
> >>> size_t length, int access);
> >>> int mlx5_dereg_mr(struct verbs_mr *mr); diff --git a/providers/mlx5/verbs.c
> >>> b/providers/mlx5/verbs.c index 44926610..129dc772 100644
> >>> --- a/providers/mlx5/verbs.c
> >>> +++ b/providers/mlx5/verbs.c
> >>> @@ -373,7 +373,7 @@ int mlx5_free_pd(struct ibv_pd *pd) }
> >>>
> >>> struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> - int acc)
> >>> + uint64_t hca_va, int acc)
> >>> {
> >>> struct mlx5_mr *mr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -385,9 +385,8 @@ struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void
> >>> *addr, size_t length,
> >>> if (!mr)
> >>> return NULL;
> >>>
> >>> - ret = ibv_cmd_reg_mr(pd, addr, length, (uintptr_t)addr, access,
> >>> - &mr->vmr, &cmd, sizeof(cmd), &resp,
> >>> - sizeof resp);
> >>> + ret = ibv_cmd_reg_mr(pd, addr, length, hca_va, access, &mr->vmr,
> >>> &cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp));
> >>> if (ret) {
> >>> mlx5_free_buf(&(mr->buf));
> >>> free(mr);
> >>> diff --git a/providers/mthca/ah.c b/providers/mthca/ah.c index
> >>> df0cb281..70fee9a6 100644
> >>> --- a/providers/mthca/ah.c
> >>> +++ b/providers/mthca/ah.c
> >>> @@ -61,7 +61,8 @@ static struct mthca_ah_page *__add_page(struct
> >>> mthca_pd *pd, int page_size, int
> >>> return NULL;
> >>> }
> >>>
> >>> - page->mr = mthca_reg_mr(&pd->ibv_pd, page->buf.buf, page_size,
> >>> 0);
> >>> + page->mr = mthca_reg_mr(&pd->ibv_pd, page->buf.buf, page_size,
> >>> + (uint64_t) page->buf.buf, 0);
> >>> if (!page->mr) {
> >>> mthca_free_buf(&page->buf);
> >>> free(page);
> >>> diff --git a/providers/mthca/mthca.h b/providers/mthca/mthca.h index
> >>> 61042de3..b7df2f73 100644
> >>> --- a/providers/mthca/mthca.h
> >>> +++ b/providers/mthca/mthca.h
> >>> @@ -280,8 +280,8 @@ int mthca_query_port(struct ibv_context *context,
> >>> uint8_t port, struct ibv_pd *mthca_alloc_pd(struct ibv_context *context);
> >>> int mthca_free_pd(struct ibv_pd *pd);
> >>>
> >>> -struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>> int mthca_dereg_mr(struct verbs_mr *mr);
> >>>
> >>> struct ibv_cq *mthca_create_cq(struct ibv_context *context, int cqe, diff --
> >>> git a/providers/mthca/verbs.c b/providers/mthca/verbs.c index
> >>> e7a1c357..99e5ec66 100644
> >>> --- a/providers/mthca/verbs.c
> >>> +++ b/providers/mthca/verbs.c
> >>> @@ -145,10 +145,10 @@ static struct ibv_mr *__mthca_reg_mr(struct
> >>> ibv_pd *pd, void *addr,
> >>> return &vmr->ibv_mr;
> >>> }
> >>>
> >>> -struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access)
> >>> +struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> - return __mthca_reg_mr(pd, addr, length, (uintptr_t) addr, access,
> >>> 0);
> >>> + return __mthca_reg_mr(pd, addr, length, hca_va, access, 0);
> >>> }
> >>>
> >>> int mthca_dereg_mr(struct verbs_mr *vmr) diff --git
> >>> a/providers/nes/nes_umain.h b/providers/nes/nes_umain.h index
> >>> b377508d..68ec807b 100644
> >>> --- a/providers/nes/nes_umain.h
> >>> +++ b/providers/nes/nes_umain.h
> >>> @@ -355,7 +355,8 @@ int nes_uquery_device(struct ibv_context *, struct
> >>> ibv_device_attr *); int nes_uquery_port(struct ibv_context *, uint8_t, struct
> >>> ibv_port_attr *); struct ibv_pd *nes_ualloc_pd(struct ibv_context *); int
> >>> nes_ufree_pd(struct ibv_pd *); -struct ibv_mr *nes_ureg_mr(struct ibv_pd
> >>> *, void *, size_t, int);
> >>> +struct ibv_mr *nes_ureg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access);
> >>> int nes_udereg_mr(struct verbs_mr *vmr); struct ibv_cq
> >>> *nes_ucreate_cq(struct ibv_context *, int, struct ibv_comp_channel *, int);
> >>> int nes_uresize_cq(struct ibv_cq *, int); diff --git
> >>> a/providers/nes/nes_uverbs.c b/providers/nes/nes_uverbs.c index
> >>> fd14849e..e5442b78 100644
> >>> --- a/providers/nes/nes_uverbs.c
> >>> +++ b/providers/nes/nes_uverbs.c
> >>> @@ -165,8 +165,8 @@ int nes_ufree_pd(struct ibv_pd *pd)
> >>> /**
> >>> * nes_ureg_mr
> >>> */
> >>> -struct ibv_mr *nes_ureg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access)
> >>> +struct ibv_mr *nes_ureg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct verbs_mr *vmr;
> >>> struct nes_ureg_mr cmd;
> >>> @@ -177,9 +177,8 @@ struct ibv_mr *nes_ureg_mr(struct ibv_pd *pd, void
> >>> *addr,
> >>> return NULL;
> >>>
> >>> cmd.reg_type = IWNES_MEMREG_TYPE_MEM;
> >>> - if (ibv_cmd_reg_mr(pd, addr, length, (uintptr_t) addr,
> >>> - access, vmr, &cmd.ibv_cmd, sizeof(cmd),
> >>> - &resp, sizeof(resp))) {
> >>> + if (ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr,
> >>> &cmd.ibv_cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp))) {
> >>> free(vmr);
> >>>
> >>> return NULL;
> >>> diff --git a/providers/ocrdma/ocrdma_main.h
> >>> b/providers/ocrdma/ocrdma_main.h index 33def78c..aadefd96 100644
> >>> --- a/providers/ocrdma/ocrdma_main.h
> >>> +++ b/providers/ocrdma/ocrdma_main.h
> >>> @@ -269,8 +269,8 @@ int ocrdma_query_device(struct ibv_context *, struct
> >>> ibv_device_attr *); int ocrdma_query_port(struct ibv_context *, uint8_t,
> >>> struct ibv_port_attr *); struct ibv_pd *ocrdma_alloc_pd(struct ibv_context
> >>> *); int ocrdma_free_pd(struct ibv_pd *); -struct ibv_mr
> >>> *ocrdma_reg_mr(struct ibv_pd *, void *, size_t,
> >>> - int ibv_access_flags);
> >>> +struct ibv_mr *ocrdma_reg_mr(struct ibv_pd *pd, void *addr, size_t len,
> >>> + uint64_t hca_va, int access);
> >>> int ocrdma_dereg_mr(struct verbs_mr *vmr);
> >>>
> >>> struct ibv_cq *ocrdma_create_cq(struct ibv_context *, int, diff --git
> >>> a/providers/ocrdma/ocrdma_verbs.c b/providers/ocrdma/ocrdma_verbs.c
> >>> index 5bd9f5ae..2c608e26 100644
> >>> --- a/providers/ocrdma/ocrdma_verbs.c
> >>> +++ b/providers/ocrdma/ocrdma_verbs.c
> >>> @@ -186,22 +186,20 @@ int ocrdma_free_pd(struct ibv_pd *ibpd)
> >>> /*
> >>> * ocrdma_reg_mr
> >>> */
> >>> -struct ibv_mr *ocrdma_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t len, int access)
> >>> +struct ibv_mr *ocrdma_reg_mr(struct ibv_pd *pd, void *addr, size_t len,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct ocrdma_mr *mr;
> >>> struct ibv_reg_mr cmd;
> >>> struct uocrdma_reg_mr_resp resp;
> >>> - uint64_t hca_va = (uintptr_t) addr;
> >>>
> >>> mr = malloc(sizeof *mr);
> >>> if (!mr)
> >>> return NULL;
> >>> bzero(mr, sizeof *mr);
> >>>
> >>> - if (ibv_cmd_reg_mr(pd, addr, len, hca_va,
> >>> - access, &mr->vmr, &cmd, sizeof(cmd),
> >>> - &resp.ibv_resp, sizeof(resp))) {
> >>> + if (ibv_cmd_reg_mr(pd, addr, len, hca_va, access, &mr->vmr, &cmd,
> >>> + sizeof(cmd), &resp.ibv_resp, sizeof(resp))) {
> >>> free(mr);
> >>> return NULL;
> >>> }
> >>> diff --git a/providers/qedr/qelr_main.h b/providers/qedr/qelr_main.h index
> >>> ef05acbc..40381e2a 100644
> >>> --- a/providers/qedr/qelr_main.h
> >>> +++ b/providers/qedr/qelr_main.h
> >>> @@ -46,8 +46,8 @@ int qelr_query_port(struct ibv_context *, uint8_t, struct
> >>> ibv_port_attr *); struct ibv_pd *qelr_alloc_pd(struct ibv_context *); int
> >>> qelr_dealloc_pd(struct ibv_pd *);
> >>>
> >>> -struct ibv_mr *qelr_reg_mr(struct ibv_pd *, void *, size_t,
> >>> - int ibv_access_flags);
> >>> +struct ibv_mr *qelr_reg_mr(struct ibv_pd *ibpd, void *addr, size_t len,
> >>> + uint64_t hca_va, int access);
> >>> int qelr_dereg_mr(struct verbs_mr *vmr);
> >>>
> >>> struct ibv_cq *qelr_create_cq(struct ibv_context *, int, diff --git
> >>> a/providers/qedr/qelr_verbs.c b/providers/qedr/qelr_verbs.c index
> >>> c5edc097..8e406bc6 100644
> >>> --- a/providers/qedr/qelr_verbs.c
> >>> +++ b/providers/qedr/qelr_verbs.c
> >>> @@ -157,8 +157,8 @@ int qelr_dealloc_pd(struct ibv_pd *ibpd)
> >>> return rc;
> >>> }
> >>>
> >>> -struct ibv_mr *qelr_reg_mr(struct ibv_pd *ibpd, void *addr,
> >>> - size_t len, int access)
> >>> +struct ibv_mr *qelr_reg_mr(struct ibv_pd *ibpd, void *addr, size_t len,
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct qelr_mr *mr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -166,17 +166,14 @@ struct ibv_mr *qelr_reg_mr(struct ibv_pd *ibpd,
> >>> void *addr,
> >>> struct qelr_pd *pd = get_qelr_pd(ibpd);
> >>> struct qelr_devctx *cxt = get_qelr_ctx(ibpd->context);
> >>>
> >>> - uint64_t hca_va = (uintptr_t) addr;
> >>> -
> >>> mr = malloc(sizeof(*mr));
> >>> if (!mr)
> >>> return NULL;
> >>>
> >>> bzero(mr, sizeof(*mr));
> >>>
> >>> - if (ibv_cmd_reg_mr(ibpd, addr, len, hca_va,
> >>> - access, &mr->vmr, &cmd, sizeof(cmd),
> >>> - &resp.ibv_resp, sizeof(resp))) {
> >>> + if (ibv_cmd_reg_mr(ibpd, addr, len, hca_va, access, &mr->vmr,
> >>> &cmd,
> >>> + sizeof(cmd), &resp.ibv_resp, sizeof(resp))) {
> >>> free(mr);
> >>> return NULL;
> >>> }
> >>> diff --git a/providers/qedr/qelr_verbs.h b/providers/qedr/qelr_verbs.h
> >>> index 26802c3a..e8c9bcb5 100644
> >>> --- a/providers/qedr/qelr_verbs.h
> >>> +++ b/providers/qedr/qelr_verbs.h
> >>> @@ -48,8 +48,8 @@ int qelr_query_port(struct ibv_context *context,
> >>> uint8_t port, struct ibv_pd *qelr_alloc_pd(struct ibv_context *context); int
> >>> qelr_dealloc_pd(struct ibv_pd *ibpd);
> >>>
> >>> -struct ibv_mr *qelr_reg_mr(struct ibv_pd *ibpd, void *addr,
> >>> - size_t len, int access);
> >>> +struct ibv_mr *qelr_reg_mr(struct ibv_pd *ibpd, void *addr, size_t len,
> >>> + uint64_t hca_va, int access);
> >>> int qelr_dereg_mr(struct verbs_mr *mr);
> >>>
> >>> struct ibv_cq *qelr_create_cq(struct ibv_context *context, int cqe, diff --git
> >>> a/providers/rxe/rxe.c b/providers/rxe/rxe.c index 909c3f7b..c0fb32e3
> >>> 100644
> >>> --- a/providers/rxe/rxe.c
> >>> +++ b/providers/rxe/rxe.c
> >>> @@ -123,7 +123,7 @@ static int rxe_dealloc_pd(struct ibv_pd *pd) }
> >>>
> >>> static struct ibv_mr *rxe_reg_mr(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> - int access)
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct verbs_mr *vmr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -134,8 +134,8 @@ static struct ibv_mr *rxe_reg_mr(struct ibv_pd *pd,
> >>> void *addr, size_t length,
> >>> if (!vmr)
> >>> return NULL;
> >>>
> >>> - ret = ibv_cmd_reg_mr(pd, addr, length, (uintptr_t)addr, access, vmr,
> >>> - &cmd, sizeof cmd, &resp, sizeof resp);
> >>> + ret = ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr, &cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp));
> >>> if (ret) {
> >>> free(vmr);
> >>> return NULL;
> >>> diff --git a/providers/vmw_pvrdma/pvrdma.h
> >>> b/providers/vmw_pvrdma/pvrdma.h index ebd50ce1..d90bd809 100644
> >>> --- a/providers/vmw_pvrdma/pvrdma.h
> >>> +++ b/providers/vmw_pvrdma/pvrdma.h
> >>> @@ -281,8 +281,8 @@ int pvrdma_query_port(struct ibv_context *context,
> >>> uint8_t port, struct ibv_pd *pvrdma_alloc_pd(struct ibv_context *context);
> >>> int pvrdma_free_pd(struct ibv_pd *pd);
> >>>
> >>> -struct ibv_mr *pvrdma_reg_mr(struct ibv_pd *pd, void *addr,
> >>> - size_t length, int access);
> >>> +struct ibv_mr *pvrdma_reg_mr(struct ibv_pd *pd, void *addr, size_t
> >>> length,
> >>> + uint64_t hca_va, int access);
> >>> int pvrdma_dereg_mr(struct verbs_mr *mr);
> >>>
> >>> struct ibv_cq *pvrdma_create_cq(struct ibv_context *context, int cqe, diff -
> >>> -git a/providers/vmw_pvrdma/verbs.c b/providers/vmw_pvrdma/verbs.c
> >>> index e27952bf..e8423c01 100644
> >>> --- a/providers/vmw_pvrdma/verbs.c
> >>> +++ b/providers/vmw_pvrdma/verbs.c
> >>> @@ -112,7 +112,7 @@ int pvrdma_free_pd(struct ibv_pd *pd) }
> >>>
> >>> struct ibv_mr *pvrdma_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> >>> - int access)
> >>> + uint64_t hca_va, int access)
> >>> {
> >>> struct verbs_mr *vmr;
> >>> struct ibv_reg_mr cmd;
> >>> @@ -123,9 +123,8 @@ struct ibv_mr *pvrdma_reg_mr(struct ibv_pd *pd,
> >>> void *addr, size_t length,
> >>> if (!vmr)
> >>> return NULL;
> >>>
> >>> - ret = ibv_cmd_reg_mr(pd, addr, length, (uintptr_t) addr,
> >>> - access, vmr, &cmd, sizeof(cmd),
> >>> - &resp, sizeof(resp));
> >>> + ret = ibv_cmd_reg_mr(pd, addr, length, hca_va, access, vmr, &cmd,
> >>> + sizeof(cmd), &resp, sizeof(resp));
> >>> if (ret) {
> >>> free(vmr);
> >>> return NULL;
> >>> --
> >>> 2.20.1
> >>
>
