On Thu, Jan 16, 2020 at 12:05:13PM +0800, Weihang Li wrote:
>
>
> On 2020/1/16 4:56, Jason Gunthorpe wrote:
> > On Wed, Jan 15, 2020 at 09:42:26AM +0800, Weihang Li wrote:
> >> From: Jiaran Zhang <zhangjiaran@xxxxxxxxxx>
> >>
> >> To support extended atomic operations including cmp & swap and fetch & add
> >> of 8 bytes, 16 bytes, 32 bytes, 64 bytes in userspace, some field in qpc
> >> should be configured.
> >>
> >> Signed-off-by: Jiaran Zhang <zhangjiaran@xxxxxxxxxx>
> >> Signed-off-by: Weihang Li <liweihang@xxxxxxxxxx>
> >> drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 16 +++++++++++++++-
> >> drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 3 ++-
> >> 2 files changed, 17 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >> index f1e0ba6..7edf3d8 100644
> >> +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >> @@ -1692,7 +1692,7 @@ static int hns_roce_v2_profile(struct hns_roce_dev *hr_dev)
> >> caps->max_srq_desc_sz = HNS_ROCE_V2_MAX_SRQ_DESC_SZ;
> >> caps->qpc_entry_sz = HNS_ROCE_V2_QPC_ENTRY_SZ;
> >> caps->irrl_entry_sz = HNS_ROCE_V2_IRRL_ENTRY_SZ;
> >> - caps->trrl_entry_sz = HNS_ROCE_V2_TRRL_ENTRY_SZ;
> >> + caps->trrl_entry_sz = HNS_ROCE_V2_EXT_ATOMIC_TRRL_ENTRY_SZ;
> >> caps->cqc_entry_sz = HNS_ROCE_V2_CQC_ENTRY_SZ;
> >> caps->srqc_entry_sz = HNS_ROCE_V2_SRQC_ENTRY_SZ;
> >> caps->mtpt_entry_sz = HNS_ROCE_V2_MTPT_ENTRY_SZ;
> >> @@ -3286,6 +3286,9 @@ static void set_access_flags(struct hns_roce_qp *hr_qp,
> >> roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
> >> !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >> roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S, 0);
> >> + roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S,
> >> + !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >> + roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S, 0);
> >> }
> >>
> >> static void set_qpc_wqe_cnt(struct hns_roce_qp *hr_qp,
> >> @@ -3653,6 +3656,12 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
> >> IB_ACCESS_REMOTE_ATOMIC));
> >> roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
> >> 0);
> >> + roce_set_bit(context->byte_76_srqn_op_en,
> >> + V2_QPC_BYTE_76_EXT_ATE_S,
> >> + !!(attr->qp_access_flags &
> >> + IB_ACCESS_REMOTE_ATOMIC));
> >> + roce_set_bit(qpc_mask->byte_76_srqn_op_en,
> >> + V2_QPC_BYTE_76_EXT_ATE_S, 0);
> >> } else {
> >> roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S,
> >> !!(hr_qp->access_flags & IB_ACCESS_REMOTE_READ));
> >> @@ -3668,6 +3677,11 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
> >> !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >> roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
> >> 0);
> >> + roce_set_bit(context->byte_76_srqn_op_en,
> >> + V2_QPC_BYTE_76_EXT_ATE_S,
> >> + !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
> >> + roce_set_bit(qpc_mask->byte_76_srqn_op_en,
> >> + V2_QPC_BYTE_76_EXT_ATE_S, 0);
> >> }
> >
> > What happens to your userspace if it runs on an old kernel and tries
> > to use extended atomic?
> >
> > Jason
> >
>
> Hi Jason,
>
> If the hns userspace runs with old kernel, the hardware will report a asynchronous
> event for the extended atomic operation and modify the qp to error state because
> the enable bit in this qp's context hasn't been set.
>
> The driver will print like this:
>
> [ 1252.240921] hns3 0000:7d:00.0: Invalid request local work queue 0x9 error.
> [ 1252.247772] hns3 0000:7d:00.0: no hr_qp can be found!
Ideally the provider will not set
IBV_PCI_ATOMIC_OPERATION_4_BYTE_SIZE_SUP and related without kernel
support..
I've applied this patch, but I feel like you may need a followup to
fix the capability reporting?
Jason
