On Sun, Jul 08, 2018 at 03:27:03PM +0300, Yuval Bason wrote:
> +int qelr_post_srq_recv(struct ibv_srq *ibsrq, struct ibv_recv_wr *wr,
> + struct ibv_recv_wr **bad_wr)
> +{
> + struct qelr_devctx *cxt = get_qelr_ctx(ibsrq->context);
> + struct qelr_srq *srq = get_qelr_srq(ibsrq);
> + struct qelr_srq_hwq_info *hw_srq = &srq->hw_srq;
> + struct qelr_chain *chain;
> + int status = 0;
> +
> + pthread_spin_lock(&srq->lock);
> +
> + chain = &srq->hw_srq.chain;
> + while (wr) {
> + struct rdma_srq_wqe_header *hdr;
> + int i;
> +
> + if (!qelr_srq_elem_left(hw_srq) ||
> + wr->num_sge > srq->hw_srq.max_sges) {
> + DP_ERR(cxt->dbg_fp,
> + "Can't post WR (%d,%d) || (%d > %d)\n",
> + hw_srq->wr_prod_cnt, hw_srq->wr_cons_cnt,
> + wr->num_sge,
> + srq->hw_srq.max_sges);
> + status = -ENOMEM;
> + *bad_wr = wr;
> + break;
> + }
> +
> + hdr = qelr_chain_produce(chain);
> +
> + /* Set number of sge and work request id in header */
> + SRQ_HDR_SET(hdr, wr->wr_id, wr->num_sge);
> +
> + /* PBL is maintained in case of WR granularity.
> + * So increment WR producer in case we post a WR.
> + */
> + hw_srq->wr_prod_cnt++;
> + hw_srq->wqe_prod++;
> + hw_srq->sge_prod++;
> +
> + DP_VERBOSE(cxt->dbg_fp, QELR_MSG_SRQ,
> + "SRQ WR: SGEs: %d with wr_id[%d] = %lx\n",
> + wr->num_sge, hw_srq->wqe_prod, wr->wr_id);
> +
> + for (i = 0; i < wr->num_sge; i++) {
> + struct rdma_srq_sge *srq_sge;
> +
> + srq_sge = qelr_chain_produce(chain);
> + /* Set SGE length, lkey and address */
> + SRQ_SGE_SET(srq_sge, wr->sg_list[i].addr,
> + wr->sg_list[i].length, wr->sg_list[i].lkey);
> +
> + DP_VERBOSE(cxt->dbg_fp, QELR_MSG_SRQ,
> + "[%d]: len %d key %x addr %x:%x\n",
> + i, srq_sge->length, srq_sge->l_key,
> + srq_sge->addr.hi, srq_sge->addr.lo);
> + hw_srq->sge_prod++;
> + }
> +
> + /* Flush WQE and SGE information before updating producer */
> + mmio_wc_start();
What is this about? mmio_wc_start() doesn't order system memory with
respect to WC writes, it is only ordering writes themselves.. If the
above code wrote to system memory then it needs a
udma_to_device_barrier() ..
> + /* SRQ producer is 8 bytes. Need to update SGE producer index
> + * in first 4 bytes and need to update WQE producer in
> + * next 4 bytes.
> + */
> +
> + struct rdma_srq_producers *virt_prod;
> +
> + virt_prod = srq->hw_srq.virt_prod_pair_addr;
> + virt_prod->sge_prod = hw_srq->sge_prod;
> + virt_prod->wqe_prod = hw_srq->wqe_prod;
> +
> + /* Flush producer after updating it. */
> + mmio_flush_writes();
This is sure funny looking.
Writes to BAR backed WC memory must be done via the mmio_write_xyz
macros, not via pointer de-ref.
If this isn't BAR backed memory then mmio_wc_start/mmio_flush_writes
cannot be used.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
