On 2020/1/16 4:56, Jason Gunthorpe wrote:
> On Wed, Jan 15, 2020 at 09:42:26AM +0800, Weihang Li wrote:
>> From: Jiaran Zhang <zhangjiaran@xxxxxxxxxx>
>>
>> To support extended atomic operations including cmp & swap and fetch & add
>> of 8 bytes, 16 bytes, 32 bytes, 64 bytes in userspace, some field in qpc
>> should be configured.
>>
>> Signed-off-by: Jiaran Zhang <zhangjiaran@xxxxxxxxxx>
>> Signed-off-by: Weihang Li <liweihang@xxxxxxxxxx>
>> drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 16 +++++++++++++++-
>> drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 3 ++-
>> 2 files changed, 17 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
>> index f1e0ba6..7edf3d8 100644
>> +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
>> @@ -1692,7 +1692,7 @@ static int hns_roce_v2_profile(struct hns_roce_dev *hr_dev)
>> caps->max_srq_desc_sz = HNS_ROCE_V2_MAX_SRQ_DESC_SZ;
>> caps->qpc_entry_sz = HNS_ROCE_V2_QPC_ENTRY_SZ;
>> caps->irrl_entry_sz = HNS_ROCE_V2_IRRL_ENTRY_SZ;
>> - caps->trrl_entry_sz = HNS_ROCE_V2_TRRL_ENTRY_SZ;
>> + caps->trrl_entry_sz = HNS_ROCE_V2_EXT_ATOMIC_TRRL_ENTRY_SZ;
>> caps->cqc_entry_sz = HNS_ROCE_V2_CQC_ENTRY_SZ;
>> caps->srqc_entry_sz = HNS_ROCE_V2_SRQC_ENTRY_SZ;
>> caps->mtpt_entry_sz = HNS_ROCE_V2_MTPT_ENTRY_SZ;
>> @@ -3286,6 +3286,9 @@ static void set_access_flags(struct hns_roce_qp *hr_qp,
>> roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
>> !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
>> roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S, 0);
>> + roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S,
>> + !!(access_flags & IB_ACCESS_REMOTE_ATOMIC));
>> + roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_EXT_ATE_S, 0);
>> }
>>
>> static void set_qpc_wqe_cnt(struct hns_roce_qp *hr_qp,
>> @@ -3653,6 +3656,12 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
>> IB_ACCESS_REMOTE_ATOMIC));
>> roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
>> 0);
>> + roce_set_bit(context->byte_76_srqn_op_en,
>> + V2_QPC_BYTE_76_EXT_ATE_S,
>> + !!(attr->qp_access_flags &
>> + IB_ACCESS_REMOTE_ATOMIC));
>> + roce_set_bit(qpc_mask->byte_76_srqn_op_en,
>> + V2_QPC_BYTE_76_EXT_ATE_S, 0);
>> } else {
>> roce_set_bit(context->byte_76_srqn_op_en, V2_QPC_BYTE_76_RRE_S,
>> !!(hr_qp->access_flags & IB_ACCESS_REMOTE_READ));
>> @@ -3668,6 +3677,11 @@ static void modify_qp_init_to_init(struct ib_qp *ibqp,
>> !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
>> roce_set_bit(qpc_mask->byte_76_srqn_op_en, V2_QPC_BYTE_76_ATE_S,
>> 0);
>> + roce_set_bit(context->byte_76_srqn_op_en,
>> + V2_QPC_BYTE_76_EXT_ATE_S,
>> + !!(hr_qp->access_flags & IB_ACCESS_REMOTE_ATOMIC));
>> + roce_set_bit(qpc_mask->byte_76_srqn_op_en,
>> + V2_QPC_BYTE_76_EXT_ATE_S, 0);
>> }
>
> What happens to your userspace if it runs on an old kernel and tries
> to use extended atomic?
>
> Jason
>
Hi Jason,
If the hns userspace runs with old kernel, the hardware will report a asynchronous
event for the extended atomic operation and modify the qp to error state because
the enable bit in this qp's context hasn't been set.
The driver will print like this:
[ 1252.240921] hns3 0000:7d:00.0: Invalid request local work queue 0x9 error.
[ 1252.247772] hns3 0000:7d:00.0: no hr_qp can be found!
Thanks
Weihang
> .
>
