On Mon, Jul 08, 2024 at 05:30:58PM +0800, Junxian Huang wrote:
>
>
> On 2024/7/8 16:59, Leon Romanovsky wrote:
> > On Mon, Jul 08, 2024 at 04:45:34PM +0800, Junxian Huang wrote:
> >>
> >>
> >> On 2024/7/8 16:27, Leon Romanovsky wrote:
> >>> On Mon, Jul 08, 2024 at 03:46:26PM +0800, Junxian Huang wrote:
> >>>>
> >>>>
> >>>> On 2024/7/8 15:33, Leon Romanovsky wrote:
> >>>>> On Mon, Jul 08, 2024 at 02:50:50PM +0800, Junxian Huang wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 2024/7/8 13:38, Leon Romanovsky wrote:
> >>>>>>> On Mon, Jul 08, 2024 at 10:29:54AM +0800, Junxian Huang wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2024/7/7 16:30, Leon Romanovsky wrote:
> >>>>>>>>> On Fri, Jul 05, 2024 at 04:59:30PM +0800, Junxian Huang wrote:
> >>>>>>>>>> From: wenglianfa <wenglianfa@xxxxxxxxxx>
> >>>>>>>>>>
> >>>>>>>>>> During reset, cmdq events won't be reported, leading to a long and
> >>>>>>>>>> unnecessary wait. Notify all the cmdqs to stop waiting at the beginning
> >>>>>>>>>> of reset.
> >>>>>>>>>>
> >>>>>>>>>> Fixes: 9a4435375cd1 ("IB/hns: Add driver files for hns RoCE driver")
> >>>>>>>>>> Signed-off-by: wenglianfa <wenglianfa@xxxxxxxxxx>
> >>>>>>>>>> Signed-off-by: Junxian Huang <huangjunxian6@xxxxxxxxxxxxx>
> >>>>>>>>>> ---
> >>>>>>>>>> drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 18 ++++++++++++++++++
> >>>>>>>>>> 1 file changed, 18 insertions(+)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >>>>>>>>>> index a5d746a5cc68..ff135df1a761 100644
> >>>>>>>>>> --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >>>>>>>>>> +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >>>>>>>>>> @@ -6977,6 +6977,21 @@ static void hns_roce_hw_v2_uninit_instance(struct hnae3_handle *handle,
> >>>>>>>>>>
> >>>>>>>>>> handle->rinfo.instance_state = HNS_ROCE_STATE_NON_INIT;
> >>>>>>>>>> }
> >>>>>>>>>> +
> >>>>>>>>>> +static void hns_roce_v2_reset_notify_cmd(struct hns_roce_dev *hr_dev)
> >>>>>>>>>> +{
> >>>>>>>>>> + struct hns_roce_cmdq *hr_cmd = &hr_dev->cmd;
> >>>>>>>>>> + int i;
> >>>>>>>>>> +
> >>>>>>>>>> + if (!hr_dev->cmd_mod)
> >>>>>>>>>
> >>>>>>>>> What prevents cmd_mod from being changed?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> It's set when the device is being initialized, and won't be changed after that.
> >>>>>>>
> >>>>>>> This is exactly the point, you are assuming that the device is already
> >>>>>>> ininitialized or not initialized at all. What prevents hns_roce_v2_reset_notify_cmd()
> >>>>>>> from being called in the middle of initialization?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>
> >>>>>> This is ensured by hns3 NIC driver.
> >>>>>>
> >>>>>> Initialization and reset of hns RoCE are both called by hns3. It will check the state
> >>>>>> of RoCE device (see line 3798), and notify RoCE device to reset (hns_roce_v2_reset_notify_cmd()
> >>>>>> is called) only if the RoCE device has been already initialized:
> >>>>>
> >>>>> So why do you have "if (!hr_dev->cmd_mod)" check in the code?
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>
> >>>> cmd_mod indicates the mode of cmdq (0: poll mode, 1: event mode).
> >>>> This patch only affects event mode because HW won't report events during reset.
> >>>
> >>> You set cmd_mod to 1 in hns_roce_hw_v2_init_instance() without any
> >>> condition, I don't see when hns v2 IB device is created and continue
> >>> to operate in polling mode.
> >>>
> >>> Thanks
> >>>
> >>
> >> Event mode is the default. In hns_roce_cmd_use_events(), if kcalloc() fails
> >> then it'll be set to polling mode instead.
> >
> > 1. Awesome, and we are returning back to the question. What prevents
> > hns_roce_v2_reset_notify_cmd() from being called in the middle of
> > changing cmd_mod from 1 to 0 and from 0 to 1?
>
> The changing of cmd_mod is during the initialization of a device. The call
> of hns_roce_v2_reset_notify_cmd() is during reset. As I said previously,
> the hns3 NIC driver ensures that there will be no concurrency between
> initialization and reset, and therefore hns_roce_v2_reset_notify_cmd() won't
> be called in the middle of changing cmd_mod.
>
> > 2. This cmd_mode swtich from 1 to 0 should be removed. Failure to
> > allocate memory is not a reason to switch to polling mode. The reason
> > can be HW limitation, but not OS limitation.
> >
>
> But event mode relies on the allocated resource. If the allocation fails and
> we don't switch to polling mode, the driver won't work any more. Are you suggesting
> we should return an error and fail the initialization in this case?
Yes, please.
Thanks
>
> Junxian
>
> > Thanks
> >
> >>
> >> Junxian
> >>
> >>>>
> >>>> Junxian
> >>>>
> >>>>>>
> >>>>>> 3791 static int hclge_notify_roce_client(struct hclge_dev *hdev,
> >>>>>> 3792 enum hnae3_reset_notify_type type)
> >>>>>> 3793 {
> >>>>>> 3794 struct hnae3_handle *handle = &hdev->vport[0].roce;
> >>>>>> 3795 struct hnae3_client *client = hdev->roce_client;
> >>>>>> 3796 int ret;
> >>>>>> 3797
> >>>>>> 3798 if (!test_bit(HCLGE_STATE_ROCE_REGISTERED, &hdev->state) || !client)
> >>>>>> 3799 return 0;
> >>>>>> 3800
> >>>>>> 3801 if (!client->ops->reset_notify)
> >>>>>> 3802 return -EOPNOTSUPP;
> >>>>>> 3803
> >>>>>> 3804 ret = client->ops->reset_notify(handle, type);
> >>>>>> 3805 if (ret)
> >>>>>> 3806 dev_err(&hdev->pdev->dev, "notify roce client failed %d(%d)",
> >>>>>> 3807 type, ret);
> >>>>>> 3808
> >>>>>> 3809 return ret;
> >>>>>> 3810 }
> >>>>>>
> >>>>>> And the bit is set (see line 11246) after the initialization has been done (line 11242):
> >>>>>>
> >>>>>> 11224 static int hclge_init_roce_client_instance(struct hnae3_ae_dev *ae_dev,
> >>>>>> 11225 struct hclge_vport *vport)
> >>>>>> 11226 {
> >>>>>> 11227 struct hclge_dev *hdev = ae_dev->priv;
> >>>>>> 11228 struct hnae3_client *client;
> >>>>>> 11229 int rst_cnt;
> >>>>>> 11230 int ret;
> >>>>>> 11231
> >>>>>> 11232 if (!hnae3_dev_roce_supported(hdev) || !hdev->roce_client ||
> >>>>>> 11233 !hdev->nic_client)
> >>>>>> 11234 return 0;
> >>>>>> 11235
> >>>>>> 11236 client = hdev->roce_client;
> >>>>>> 11237 ret = hclge_init_roce_base_info(vport);
> >>>>>> 11238 if (ret)
> >>>>>> 11239 return ret;
> >>>>>> 11240
> >>>>>> 11241 rst_cnt = hdev->rst_stats.reset_cnt;
> >>>>>> 11242 ret = client->ops->init_instance(&vport->roce);
> >>>>>> 11243 if (ret)
> >>>>>> 11244 return ret;
> >>>>>> 11245
> >>>>>> 11246 set_bit(HCLGE_STATE_ROCE_REGISTERED, &hdev->state);
> >>>>>> 11247 if (test_bit(HCLGE_STATE_RST_HANDLING, &hdev->state) ||
> >>>>>> 11248 rst_cnt != hdev->rst_stats.reset_cnt) {
> >>>>>> 11249 ret = -EBUSY;
> >>>>>> 11250 goto init_roce_err;
> >>>>>> 11251 }
> >>>>>>
> >>>>>> Junxian
> >>>>>>
> >>>>>>>>
> >>>>>>>> Junxian
> >>>>>>>>
> >>>>>>>>>> + return;
> >>>>>>>>>> +
> >>>>>>>>>> + for (i = 0; i < hr_cmd->max_cmds; i++) {
> >>>>>>>>>> + hr_cmd->context[i].result = -EBUSY;
> >>>>>>>>>> + complete(&hr_cmd->context[i].done);
> >>>>>>>>>> + }
> >>>>>>>>>> +}
> >>>>>>>>>> +
> >>>>>>>>>> static int hns_roce_hw_v2_reset_notify_down(struct hnae3_handle *handle)
> >>>>>>>>>> {
> >>>>>>>>>> struct hns_roce_dev *hr_dev;
> >>>>>>>>>> @@ -6997,6 +7012,9 @@ static int hns_roce_hw_v2_reset_notify_down(struct hnae3_handle *handle)
> >>>>>>>>>> hr_dev->dis_db = true;
> >>>>>>>>>> hr_dev->state = HNS_ROCE_DEVICE_STATE_RST_DOWN;
> >>>>>>>>>>
> >>>>>>>>>> + /* Complete the CMDQ event in advance during the reset. */
> >>>>>>>>>> + hns_roce_v2_reset_notify_cmd(hr_dev);
> >>>>>>>>>> +
> >>>>>>>>>> return 0;
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> 2.33.0
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>
> >
>
