> -----Original Message----- > From: Jason Gunthorpe [mailto:jgg@xxxxxxxxxxxx] > Sent: Monday, July 9, 2018 8:18 PM > To: Bason, Yuval <Yuval.Bason@xxxxxxxxxx> > Cc: dledford@xxxxxxxxxx; leon@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; > Kalderon, Michal <Michal.Kalderon@xxxxxxxxxx> > Subject: Re: [PATCH rdma-core] libqedr: add support for SRQ's > > External Email > > On Sun, Jul 08, 2018 at 03:27:03PM +0300, Yuval Bason wrote: > > +int qelr_post_srq_recv(struct ibv_srq *ibsrq, struct ibv_recv_wr *wr, > > + struct ibv_recv_wr **bad_wr) { > > + struct qelr_devctx *cxt = get_qelr_ctx(ibsrq->context); > > + struct qelr_srq *srq = get_qelr_srq(ibsrq); > > + struct qelr_srq_hwq_info *hw_srq = &srq->hw_srq; > > + struct qelr_chain *chain; > > + int status = 0; > > + > > + pthread_spin_lock(&srq->lock); > > + > > + chain = &srq->hw_srq.chain; > > + while (wr) { > > + struct rdma_srq_wqe_header *hdr; > > + int i; > > + > > + if (!qelr_srq_elem_left(hw_srq) || > > + wr->num_sge > srq->hw_srq.max_sges) { > > + DP_ERR(cxt->dbg_fp, > > + "Can't post WR (%d,%d) || (%d > %d)\n", > > + hw_srq->wr_prod_cnt, hw_srq->wr_cons_cnt, > > + wr->num_sge, > > + srq->hw_srq.max_sges); > > + status = -ENOMEM; > > + *bad_wr = wr; > > + break; > > + } > > + > > + hdr = qelr_chain_produce(chain); > > + > > + /* Set number of sge and work request id in header */ > > + SRQ_HDR_SET(hdr, wr->wr_id, wr->num_sge); > > + > > + /* PBL is maintained in case of WR granularity. > > + * So increment WR producer in case we post a WR. > > + */ > > + hw_srq->wr_prod_cnt++; > > + hw_srq->wqe_prod++; > > + hw_srq->sge_prod++; > > + > > + DP_VERBOSE(cxt->dbg_fp, QELR_MSG_SRQ, > > + "SRQ WR: SGEs: %d with wr_id[%d] = %lx\n", > > + wr->num_sge, hw_srq->wqe_prod, wr->wr_id); > > + > > + for (i = 0; i < wr->num_sge; i++) { > > + struct rdma_srq_sge *srq_sge; > > + > > + srq_sge = qelr_chain_produce(chain); > > + /* Set SGE length, lkey and address */ > > + SRQ_SGE_SET(srq_sge, wr->sg_list[i].addr, > > + wr->sg_list[i].length, > > + wr->sg_list[i].lkey); > > + > > + DP_VERBOSE(cxt->dbg_fp, QELR_MSG_SRQ, > > + "[%d]: len %d key %x addr %x:%x\n", > > + i, srq_sge->length, srq_sge->l_key, > > + srq_sge->addr.hi, srq_sge->addr.lo); > > + hw_srq->sge_prod++; > > + } > > + > > + /* Flush WQE and SGE information before updating producer */ > > + mmio_wc_start(); > > What is this about? mmio_wc_start() doesn't order system memory with > respect to WC writes, it is only ordering writes themselves.. If the above > code wrote to system memory then it needs a > udma_to_device_barrier() .. Thanks for pointing out. Will fix. > > > + /* SRQ producer is 8 bytes. Need to update SGE producer index > > + * in first 4 bytes and need to update WQE producer in > > + * next 4 bytes. > > + */ > > + > > + struct rdma_srq_producers *virt_prod; > > + > > + virt_prod = srq->hw_srq.virt_prod_pair_addr; > > + virt_prod->sge_prod = hw_srq->sge_prod; > > + virt_prod->wqe_prod = hw_srq->wqe_prod; > > + > > + /* Flush producer after updating it. */ > > + mmio_flush_writes(); > > This is sure funny looking. > > Writes to BAR backed WC memory must be done via the mmio_write_xyz > macros, not via pointer de-ref. > > If this isn't BAR backed memory then mmio_wc_start/mmio_flush_writes > cannot be used. Absolutely. This isn't BAR backed memory. Will fix to use the correct barriers. > > Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
