On Wed, Jun 20, 2018 at 04:19:41PM +0000, Parav Pandit wrote: > > > > -----Original Message----- > > From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma- > > owner@xxxxxxxxxxxxxxx] On Behalf Of Vijay Immanuel > > Sent: Monday, June 18, 2018 8:47 PM > > To: linux-rdma@xxxxxxxxxxxxxxx > > Subject: [PATCH] IB/rxe: vary the source udp port for receive scaling > > > > Select the source udp port number for a QP based on a hash of the source and > > destination QPNs. This provides a better spread of traffic across NIC RX queues. > > > > Signed-off-by: Vijay Immanuel <vijayi@xxxxxxxxxxxxxxxxx> > > --- > > drivers/infiniband/sw/rxe/rxe_net.c | 6 ++---- > > drivers/infiniband/sw/rxe/rxe_qp.c | 8 +++++++- > > drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + > > 3 files changed, 10 insertions(+), 5 deletions(-) > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_net.c > > b/drivers/infiniband/sw/rxe/rxe_net.c > > index 9da6e37..04e5097 100644 > > --- a/drivers/infiniband/sw/rxe/rxe_net.c > > +++ b/drivers/infiniband/sw/rxe/rxe_net.c > > @@ -409,8 +409,7 @@ static int prepare4(struct rxe_dev *rxe, struct > > rxe_pkt_info *pkt, > > if (!memcmp(saddr, daddr, sizeof(*daddr))) > > pkt->mask |= RXE_LOOPBACK_MASK; > > > > - prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT), > > - htons(ROCE_V2_UDP_DPORT)); > > + prepare_udp_hdr(skb, htons(qp->src_port), > > htons(ROCE_V2_UDP_DPORT)); > > > > prepare_ipv4_hdr(dst, skb, saddr->s_addr, daddr->s_addr, > > IPPROTO_UDP, > > av->grh.traffic_class, av->grh.hop_limit, df, xnet); @@ > > -440,8 +439,7 @@ static int prepare6(struct rxe_dev *rxe, struct rxe_pkt_info > > *pkt, > > if (!memcmp(saddr, daddr, sizeof(*daddr))) > > pkt->mask |= RXE_LOOPBACK_MASK; > > > > - prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT), > > - htons(ROCE_V2_UDP_DPORT)); > > + prepare_udp_hdr(skb, htons(qp->src_port), > > htons(ROCE_V2_UDP_DPORT)); > > > > prepare_ipv6_hdr(dst, skb, saddr, daddr, IPPROTO_UDP, > > av->grh.traffic_class, > > diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c > > b/drivers/infiniband/sw/rxe/rxe_qp.c > > index b9f7aa1..fa8bfa9 100644 > > --- a/drivers/infiniband/sw/rxe/rxe_qp.c > > +++ b/drivers/infiniband/sw/rxe/rxe_qp.c > > @@ -227,6 +227,8 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct > > rxe_qp *qp, > > return err; > > qp->sk->sk->sk_user_data = qp; > > > > + qp->src_port = RXE_ROCE_V2_SPORT; > > + > QP shouldn't be transmitting anything until it is in RTR state. So this initialization doesn't seem necessary. I think it's necessary for UD. The dest QP may not be set via the QP attr, so this initialization makes sure we're using an ephemeral port. > > > qp->sq.max_wr = init->cap.max_send_wr; > > qp->sq.max_sge = init->cap.max_send_sge; > > qp->sq.max_inline = init->cap.max_inline_data; > > @@ -706,8 +708,12 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct > > ib_qp_attr *attr, int mask, > > if (mask & IB_QP_PATH_MIG_STATE) > > qp->attr.path_mig_state = attr->path_mig_state; > > > > - if (mask & IB_QP_DEST_QPN) > > + if (mask & IB_QP_DEST_QPN) { > > qp->attr.dest_qp_num = attr->dest_qp_num; > > + qp->src_port = RXE_ROCE_V2_SPORT + > > + (hash_64_generic(((u64)attr->dest_qp_num << 24) + > > + (u64)qp_num(qp), 14) & 0x3fff); > > + } > > > > if (mask & IB_QP_STATE) { > > qp->attr.qp_state = attr->qp_state; > > diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h > > b/drivers/infiniband/sw/rxe/rxe_verbs.h > > index af1470d..0d9200878 100644 > > --- a/drivers/infiniband/sw/rxe/rxe_verbs.h > > +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h > > @@ -248,6 +248,7 @@ struct rxe_qp { > > > > struct socket *sk; > > u32 dst_cookie; > > + u16 src_port; > > > > struct rxe_av pri_av; > > struct rxe_av alt_av; > > -- > > 2.7.4 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body > > of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at > > http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
