> -----Original Message-----
> From: Long Li <longli@xxxxxxxxxxxxx>
> Sent: Friday, June 9, 2023 1:48 AM
> To: Wei Hu <weh@xxxxxxxxxxxxx>; netdev@xxxxxxxxxxxxxxx; linux-
> hyperv@xxxxxxxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; Ajay Sharma
> <sharmaajay@xxxxxxxxxxxxx>; jgg@xxxxxxxx; leon@xxxxxxxxxx; KY
> Srinivasan <kys@xxxxxxxxxxxxx>; Haiyang Zhang <haiyangz@xxxxxxxxxxxxx>;
> wei.liu@xxxxxxxxxx; Dexuan Cui <decui@xxxxxxxxxxxxx>;
> davem@xxxxxxxxxxxxx; edumazet@xxxxxxxxxx; kuba@xxxxxxxxxx;
> pabeni@xxxxxxxxxx; vkuznets@xxxxxxxxxx; ssengar@xxxxxxxxxxxxxxxxxxx;
> shradhagupta@xxxxxxxxxxxxxxxxxxx
> Subject: RE: [PATCH v2 1/1] RDMA/mana_ib: Add EQ interrupt support to
> mana ib driver.
>
> > Subject: RE: [PATCH v2 1/1] RDMA/mana_ib: Add EQ interrupt support to
> > mana ib driver.
> >
> >
> >
> > > -----Original Message-----
> > > Subject: RE: [PATCH v2 1/1] RDMA/mana_ib: Add EQ interrupt support
> > > to mana ib driver.
> > >
> > > > Subject: [PATCH v2 1/1] RDMA/mana_ib: Add EQ interrupt support to
> > > > mana ib driver.
> > > >
> > > > Add EQ interrupt support for mana ib driver. Allocate EQs per
> > > > ucontext to receive interrupt. Attach EQ when CQ is created. Call
> > > > CQ interrupt handler when completion interrupt happens. EQs are
> > > > destroyed when
> > > ucontext is deallocated.
> > > >
> > > > The change calls some public APIs in mana ethernet driver to
> > > > allocate EQs and other resources. Ehe EQ process routine is also
> > > > shared by mana ethernet and mana ib drivers.
> > > >
> > > > Co-developed-by: Ajay Sharma <sharmaajay@xxxxxxxxxxxxx>
> > > > Signed-off-by: Ajay Sharma <sharmaajay@xxxxxxxxxxxxx>
> > > > Signed-off-by: Wei Hu <weh@xxxxxxxxxxxxx>
> > > > ---
> > > >
> > > > v2: Use ibdev_dbg to print error messages and return -ENOMEN
> > > > when kzalloc fails.
> > > >
> > > > drivers/infiniband/hw/mana/cq.c | 32 ++++-
> > > > drivers/infiniband/hw/mana/main.c | 87 ++++++++++++
> > > > drivers/infiniband/hw/mana/mana_ib.h | 4 +
> > > > drivers/infiniband/hw/mana/qp.c | 90 +++++++++++-
> > > > .../net/ethernet/microsoft/mana/gdma_main.c | 131 ++++++++++---
> ----
> > -
> > > > drivers/net/ethernet/microsoft/mana/mana_en.c | 1 +
> > > > include/net/mana/gdma.h | 9 +-
> > > > 7 files changed, 290 insertions(+), 64 deletions(-)
> > > >
> > > > diff --git a/drivers/infiniband/hw/mana/cq.c
> > > > b/drivers/infiniband/hw/mana/cq.c index
> d141cab8a1e6..3cd680e0e753
> > > > 100644
> > > > --- a/drivers/infiniband/hw/mana/cq.c
> > > > +++ b/drivers/infiniband/hw/mana/cq.c
> > > > @@ -12,13 +12,20 @@ int mana_ib_create_cq(struct ib_cq *ibcq,
> > > > const struct ib_cq_init_attr *attr,
> > > > struct ib_device *ibdev = ibcq->device;
> > > > struct mana_ib_create_cq ucmd = {};
> > > > struct mana_ib_dev *mdev;
> > > > + struct gdma_context *gc;
> > > > + struct gdma_dev *gd;
> > > > int err;
> > > >
> > > > mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);
> > > > + gd = mdev->gdma_dev;
> > > > + gc = gd->gdma_context;
> > > >
> > > > if (udata->inlen < sizeof(ucmd))
> > > > return -EINVAL;
> > > >
> > > > + cq->comp_vector = attr->comp_vector > gc->max_num_queues ?
> > > > + 0 : attr->comp_vector;
> > > > +
> > > > err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata-
> > > > >inlen));
> > > > if (err) {
> > > > ibdev_dbg(ibdev,
> > > > @@ -69,11 +76,32 @@ int mana_ib_destroy_cq(struct ib_cq *ibcq,
> > > > struct ib_udata *udata)
> > > > struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq);
> > > > struct ib_device *ibdev = ibcq->device;
> > > > struct mana_ib_dev *mdev;
> > > > + struct gdma_context *gc;
> > > > + struct gdma_dev *gd;
> > > > +
> > > >
> > > > mdev = container_of(ibdev, struct mana_ib_dev, ib_dev);
> > > > + gd = mdev->gdma_dev;
> > > > + gc = gd->gdma_context;
> > > >
> > > > - mana_ib_gd_destroy_dma_region(mdev, cq->gdma_region);
> > > > - ib_umem_release(cq->umem);
> > > > +
> > > > +
> > > > + if (atomic_read(&ibcq->usecnt) == 0) {
> > > > + mana_ib_gd_destroy_dma_region(mdev, cq->gdma_region);
> > >
> > > Need to check if this function fails. The following code will call
> > > kfree(gc-
> > > >cq_table[cq->id]), it's possible that IRQ is happening at the same
> > > >time if CQ
> > > is not destroyed.
> > >
> >
> > Sure. Will update.
> >
> > > > + ibdev_dbg(ibdev, "freeing gdma cq %p\n", gc->cq_table[cq-
> > > >id]);
> > > > + kfree(gc->cq_table[cq->id]);
> > > > + gc->cq_table[cq->id] = NULL;
> > > > + ib_umem_release(cq->umem);
> > > > + }
> > > >
> > > > return 0;
> > > > }
> > > > +
> > > > +void mana_ib_cq_handler(void *ctx, struct gdma_queue *gdma_cq) {
> > > > + struct mana_ib_cq *cq = ctx;
> > > > + struct ib_device *ibdev = cq->ibcq.device;
> > > > +
> > > > + ibdev_dbg(ibdev, "Enter %s %d\n", __func__, __LINE__);
> > >
> > > This debug message seems overkill?
> > >
> > > > + cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context); }
> > > > diff --git a/drivers/infiniband/hw/mana/main.c
> > > > b/drivers/infiniband/hw/mana/main.c
> > > > index 7be4c3adb4e2..e4efbcaed10e 100644
> > > > --- a/drivers/infiniband/hw/mana/main.c
> > > > +++ b/drivers/infiniband/hw/mana/main.c
> > > > @@ -143,6 +143,81 @@ int mana_ib_dealloc_pd(struct ib_pd *ibpd,
> > > > struct ib_udata *udata)
> > > > return err;
> > > > }
> > > >
> > > > +static void mana_ib_destroy_eq(struct mana_ib_ucontext *ucontext,
> > > > + struct mana_ib_dev *mdev) {
> > > > + struct gdma_context *gc = mdev->gdma_dev->gdma_context;
> > > > + struct ib_device *ibdev = ucontext->ibucontext.device;
> > > > + struct gdma_queue *eq;
> > > > + int i;
> > > > +
> > > > + if (!ucontext->eqs)
> > > > + return;
> > > > +
> > > > + for (i = 0; i < gc->max_num_queues; i++) {
> > > > + eq = ucontext->eqs[i].eq;
> > > > + if (!eq)
> > > > + continue;
> > > > +
> > > > + mana_gd_destroy_queue(gc, eq);
> > > > + }
> > > > +
> > > > + kfree(ucontext->eqs);
> > > > + ucontext->eqs = NULL;
> > > > +
> > > > + ibdev_dbg(ibdev, "destroyed eq's count %d\n", gc-
> > > >max_num_queues); }
> > >
> > > Will gc->max_num_queues change after destroying a EQ?
> > >
> >
> > I think it will not change. Also the compiler might optimize the code
> > to just read the value once and store it in a register.
> >
> > Thanks,
> > Wei
>
> This message is confusing. How about changing it to " destroyed eq.
> Maximum count %d", or just remove the count as it's not informational.
>
> Long
Sure. I will remove the count from the message.
Thanks,
Wei
