On Mon, Jul 15, 2019 at 05:13:28PM +0300, Yuval Shaia wrote:
> The virtual address that is registered is used as a base for any address
> passed later in post_recv and post_send operations.
>
> On some virtualized environment this is not correct.
>
> A guest cannot register its memory so hypervisor maps the guest physical
> address to a host virtual address and register it with the HW. Later on,
> at datapath phase, the guest fills the SGEs with addresses from its
> address space.
> Since HW cannot access guest virtual address space an extra translation
> is needed to map those addresses to be based on the host virtual address
> that was registered with the HW.
> This datapath interference affects performances.
>
> To avoid this, a logical separation between the address that is
> registered and the address that is used as a offset at datapath phase is
> needed.
> This separation is already implemented in the lower layer part
> (ibv_cmd_reg_mr) but blocked at the API level.
>
> Fix it by introducing a new API function which accepts an address from
> guest virtual address space as well, to be used as offset for later
> datapath operations.
>
> Signed-off-by: Yuval Shaia <yuval.shaia@xxxxxxxxxx>
> Reviewed-by: Leon Romanovsky <leonro@xxxxxxxxxxxx>
> v0 -> v1:
> * Change reg_mr callback signature instead of adding new callback
> * Add the new API to libibverbs/libibverbs.map.in
> v1 -> v2:
> * Do not modify reg_mr signature for version 1.0
> * Add note to man page
> v2 -> v3:
> * Rename function to reg_mr_iova (and arg-name to iova)
> * Some checkpatch issues not related to this fix but detected now
> * s/__FUNCTION__/__func
> * WARNING: function definition argument 'void *' should
> also have an identifier name
> v3 -> v4:
> * Fix commit message as suggested by Adit Ranadiv
> * Add support for efa
> v4 -> v5:
> * Update PABI
> * Update debian files
> v5 -> v6:
> * Move the new API to section in libibverbs/libibverbs.map.in
> (IBVERBS_1.7) as pointed out by Mark Haywood
> v6 -> v7:
> *
> v7 -> v8:
> * Update also redhat and suse specfiles so now all CI checks in
> github passed.
> * Leon, i have your r-b from v5, appriciate if you can take a look
> again now, with all the latest changes
> CMakeLists.txt | 4 ++--
> buildlib/cbuild | 6 ++++++
> debian/control | 2 +-
> debian/libibverbs1.symbols | 4 +++-
> libibverbs/CMakeLists.txt | 2 +-
> libibverbs/driver.h | 2 +-
> libibverbs/dummy_ops.c | 2 +-
> libibverbs/libibverbs.map.in | 5 +++++
> libibverbs/man/ibv_reg_mr.3 | 15 +++++++++++++--
> libibverbs/verbs.c | 23 ++++++++++++++++++++++-
> libibverbs/verbs.h | 7 +++++++
> providers/bnxt_re/verbs.c | 6 +++---
> providers/bnxt_re/verbs.h | 2 +-
> providers/cxgb3/iwch.h | 4 ++--
> providers/cxgb3/verbs.c | 15 +++++----------
> providers/cxgb4/libcxgb4.h | 4 ++--
> providers/cxgb4/verbs.c | 15 +++++----------
> providers/efa/verbs.c | 4 ++--
> providers/efa/verbs.h | 2 +-
> providers/hfi1verbs/hfiverbs.h | 4 ++--
> providers/hfi1verbs/verbs.c | 8 ++++----
> providers/hns/hns_roce_u.h | 2 +-
> providers/hns/hns_roce_u_verbs.c | 6 +++---
> providers/i40iw/i40iw_umain.h | 3 ++-
> providers/i40iw/i40iw_uverbs.c | 8 ++++----
> providers/ipathverbs/ipathverbs.h | 4 ++--
> providers/ipathverbs/verbs.c | 8 ++++----
> providers/mlx4/mlx4.h | 4 ++--
> providers/mlx4/verbs.c | 7 +++----
> providers/mlx5/mlx5.h | 4 ++--
> providers/mlx5/verbs.c | 7 +++----
> providers/mthca/ah.c | 3 ++-
> providers/mthca/mthca.h | 4 ++--
> providers/mthca/verbs.c | 6 +++---
> providers/nes/nes_umain.h | 3 ++-
> providers/nes/nes_uverbs.c | 9 ++++-----
> providers/ocrdma/ocrdma_main.h | 4 ++--
> providers/ocrdma/ocrdma_verbs.c | 10 ++++------
> providers/qedr/qelr_main.h | 4 ++--
> providers/qedr/qelr_verbs.c | 11 ++++-------
> providers/qedr/qelr_verbs.h | 4 ++--
> providers/rxe/rxe.c | 6 +++---
> providers/siw/siw.c | 4 ++--
> providers/vmw_pvrdma/pvrdma.h | 4 ++--
> providers/vmw_pvrdma/verbs.c | 7 +++----
> redhat/rdma-core.spec | 2 +-
> suse/rdma-core.spec | 2 +-
> 47 files changed, 154 insertions(+), 118 deletions(-)
>
> diff --git a/CMakeLists.txt b/CMakeLists.txt
> index b2613284..67112ae3 100644
> +++ b/CMakeLists.txt
> @@ -68,11 +68,11 @@ endif()
> set(PACKAGE_NAME "RDMA")
>
> # See Documentation/versioning.md
> -set(PACKAGE_VERSION "25.0")
> +set(PACKAGE_VERSION "26.0")
> # When this is changed the values in these files need changing too:
> # debian/control
> # debian/libibverbs1.symbols
> -set(IBVERBS_PABI_VERSION "25")
> +set(IBVERBS_PABI_VERSION "26")
> set(IBVERBS_PROVIDER_SUFFIX "-rdmav${IBVERBS_PABI_VERSION}.so")
'25' is still the current release-in progress.
Jason
