On Thu, May 03, 2018 at 03:19:27PM +0000, Rohit Zambre wrote:
> An independent communication path is one that shares no hardware resources
> with other communication paths. From a Verbs perspective, an independent
> path is the one obtained by the first QP in a context. The next QPs of the
> context may or may not share hardware resources amongst themselves; the
> mapping of the resources to the QPs is provider-specific. Sharing resources
> can hurt throughput in certain cases. When only one thread uses the
> independent path, we term it an uncontended independent path.
>
> Today, the user has no way to request for an independent path for an
> arbitrary QP within a context. To create multiple independent paths, the
> Verbs user must create mulitple contexts with 1 QP per context. However,
> this translates to significant hardware-resource wastage: 89% in the case
> of the ConnectX-4 mlx5 device.
>
> This RFC patch allows the user to request for uncontended independent
> communication paths in Verbs through an "independent" flag during Thread
> Domain (TD) creation. The patch also provides a first-draft implementation
> of uncontended independent paths in the mlx5 provider.
>
> In mlx5, every even-odd pair of TDs share the same UAR page, which is not
> case when the user creates multiple contexts with one TD per context. When
> the user requests for an independent TD, the driver will dynamically
> allocate a new UAR page and map bfreg_0 of that UAR to the TD. bfreg_1 of
> the UAR belonging to an independent TD is never used and is essentially
> wasted. Hence, there must be a maximum number of independent paths allowed
> within a context since the hardware resources are limited. This would be
> half of the maximum number of dynamic UARs allowed per context.
>
> Signed-off-by: Rohit Zambre <rzambre@xxxxxxx>
> libibverbs/verbs.h | 1 +
> providers/mlx5/mlx5.c | 3 +++
> providers/mlx5/mlx5.h | 2 ++
> providers/mlx5/verbs.c | 51 +++++++++++++++++++++++++++++++++++---------------
> 4 files changed, 42 insertions(+), 15 deletions(-)
>
> diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
> index eb57824..b5fa56f 100644
> +++ b/libibverbs/verbs.h
> @@ -561,6 +561,7 @@ struct ibv_pd {
> };
>
> struct ibv_td_init_attr {
> + int independent;
> uint32_t comp_mask;
> };
This isn't OK, it breaks the ABI. Needs to be after comp_mask and you
need to introduce a comp mask flag that says the data is present.
Also needs a man page update to describe the new functionality.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
