On Wed, 3 Jul 2019 09:24:54 +0800
Yanjun Zhu <yanjun.zhu@xxxxxxxxxx> wrote:
> On 2019/6/27 22:06, Konstantin Taranov wrote:
> > Make softRoce to calculate correct byte_len on receiving side when work completion
> > is generated with IB_WC_RECV_RDMA_WITH_IMM opcode.
> >
> > According to documentation byte_len must indicate the number of written
> > bytes, whereas it was always equal to zero for IB_WC_RECV_RDMA_WITH_IMM opcode.
>
> With roce NIC, what is the byte_len? Thanks a lot.
byte_len is a field of a work completion (struct ib_uverbs_wc or struct ibv_wc). It is defined in verbs and stores
the number of written bytes to the destination memory. In case of IB_WC_RECV_RDMA_WITH_IMM
completion event, the field byte_len must store the number of written bytes for incoming
RDMA_WRITE_WITH_IMM request.
>
> Zhu Yanjun
>
> >
> > The patch proposes to remember the length of an RDMA request from the RETH header, and use it
> > as byte_len when the work completion with IB_WC_RECV_RDMA_WITH_IMM opcode is generated.
> >
> > Signed-off-by: Konstantin Taranov <konstantin.taranov@xxxxxxxxxxx>
> > ---
> > drivers/infiniband/sw/rxe/rxe_resp.c | 5 ++++-
> > drivers/infiniband/sw/rxe/rxe_verbs.h | 1 +
> > 2 files changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> > index aca9f60f9b21..1cbfbd98eb22 100644
> > --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> > +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> > @@ -431,6 +431,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
> > qp->resp.va = reth_va(pkt);
> > qp->resp.rkey = reth_rkey(pkt);
> > qp->resp.resid = reth_len(pkt);
> > + qp->resp.length = reth_len(pkt);
> > }
> > access = (pkt->mask & RXE_READ_MASK) ? IB_ACCESS_REMOTE_READ
> > : IB_ACCESS_REMOTE_WRITE;
> > @@ -856,7 +857,9 @@ static enum resp_states do_complete(struct rxe_qp *qp,
> > pkt->mask & RXE_WRITE_MASK) ?
> > IB_WC_RECV_RDMA_WITH_IMM : IB_WC_RECV;
> > wc->vendor_err = 0;
> > - wc->byte_len = wqe->dma.length - wqe->dma.resid;
> > + wc->byte_len = (pkt->mask & RXE_IMMDT_MASK &&
> > + pkt->mask & RXE_WRITE_MASK) ?
> > + qp->resp.length : wqe->dma.length - wqe->dma.resid;
> >
> > /* fields after byte_len are different between kernel and user
> > * space
> > diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h
> > index e8be7f44e3be..28bfb3ece104 100644
> > --- a/drivers/infiniband/sw/rxe/rxe_verbs.h
> > +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h
> > @@ -213,6 +213,7 @@ struct rxe_resp_info {
> > struct rxe_mem *mr;
> > u32 resid;
> > u32 rkey;
> > + u32 length;
> > u64 atomic_orig;
> >
> > /* SRQ only */
