On Thu, Mar 31, 2016 at 07:03:25PM +0300, Sagi Grimberg wrote:
> mlx5 devices (Connect-IB, ConnectX-4, ConnectX-4-LX) has a limitation
> where rdma read work queue entries cannot exceed 512 bytes.
> A rdma_read wqe needs to fit in 512 bytes:
> - wqe control segment (16 bytes)
> - rdma segment (16 bytes)
> - scatter elements (16 bytes each)
>
> So max_sge_rd should be: (512 - 16 - 16) / 16 = 30.
>
> Reported-by: Christoph Hellwig <hch@xxxxxx>
> Tested-by: Christoph Hellwig <hch@xxxxxx>
> Signed-off-by: Sagi Grimberg <sagig@xxxxxxxxxxx>
Thanks Sagi,
Excellent catch, the same as in commit: a5e4ba334e2
("mlx4: Expose correct max_sge_rd limit")
Signed-off-by: Leon Romanovsky <leonro@xxxxxxxxxxxx>
> ---
> drivers/infiniband/hw/mlx5/main.c | 2 +-
> include/linux/mlx5/device.h | 11 +++++++++++
> 2 files changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> index 03c418ccbc98..ed9cefa1f6f1 100644
> --- a/drivers/infiniband/hw/mlx5/main.c
> +++ b/drivers/infiniband/hw/mlx5/main.c
> @@ -517,7 +517,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
> sizeof(struct mlx5_wqe_ctrl_seg)) /
> sizeof(struct mlx5_wqe_data_seg);
> props->max_sge = min(max_rq_sg, max_sq_sg);
> - props->max_sge_rd = props->max_sge;
> + props->max_sge_rd = MLX5_MAX_SGE_RD;
> props->max_cq = 1 << MLX5_CAP_GEN(mdev, log_max_cq);
> props->max_cqe = (1 << MLX5_CAP_GEN(mdev, log_max_cq_sz)) - 1;
> props->max_mr = 1 << MLX5_CAP_GEN(mdev, log_max_mkey);
> diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
> index 987764afa65c..f8b83792939b 100644
> --- a/include/linux/mlx5/device.h
> +++ b/include/linux/mlx5/device.h
> @@ -363,6 +363,17 @@ enum {
> MLX5_CAP_OFF_CMDIF_CSUM = 46,
> };
>
> +enum {
> + /*
> + * Max wqe size for rdma read is 512 bytes, so this
> + * limits our max_sge_rd as the wqe needs to fit:
> + * - ctrl segment (16 bytes)
> + * - rdma segment (16 bytes)
> + * - scatter elements (16 bytes each)
> + */
> + MLX5_MAX_SGE_RD = (512 - 16 - 16) / 16
> +};
> +
> struct mlx5_inbox_hdr {
> __be16 opcode;
> u8 rsvd[4];
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
