On Wed, Jun 20, 2018 at 10:12:09PM +0000, Parav Pandit wrote: > > > > -----Original Message----- > > From: Vijay Immanuel [mailto:vijayi@xxxxxxxxxxxxxxxxx] > > Sent: Wednesday, June 20, 2018 4:31 PM > > To: Parav Pandit <parav@xxxxxxxxxxxx> > > Cc: linux-rdma@xxxxxxxxxxxxxxx > > Subject: Re: [PATCH] IB/rxe: vary the source udp port for receive scaling > > > > On Wed, Jun 20, 2018 at 04:19:41PM +0000, Parav Pandit wrote: > > > > > > > > > > -----Original Message----- > > > > From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma- > > > > owner@xxxxxxxxxxxxxxx] On Behalf Of Vijay Immanuel > > > > Sent: Monday, June 18, 2018 8:47 PM > > > > To: linux-rdma@xxxxxxxxxxxxxxx > > > > Subject: [PATCH] IB/rxe: vary the source udp port for receive > > > > scaling > > > > > > > > Select the source udp port number for a QP based on a hash of the > > > > source and destination QPNs. This provides a better spread of traffic across > > NIC RX queues. > > > > > > > > Signed-off-by: Vijay Immanuel <vijayi@xxxxxxxxxxxxxxxxx> > > > > --- > > > > drivers/infiniband/sw/rxe/rxe_net.c | 6 ++---- > > > > drivers/infiniband/sw/rxe/rxe_qp.c | 8 +++++++- > > > > drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + > > > > 3 files changed, 10 insertions(+), 5 deletions(-) > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_net.c > > > > b/drivers/infiniband/sw/rxe/rxe_net.c > > > > index 9da6e37..04e5097 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe_net.c > > > > +++ b/drivers/infiniband/sw/rxe/rxe_net.c > > > > @@ -409,8 +409,7 @@ static int prepare4(struct rxe_dev *rxe, struct > > > > rxe_pkt_info *pkt, > > > > if (!memcmp(saddr, daddr, sizeof(*daddr))) > > > > pkt->mask |= RXE_LOOPBACK_MASK; > > > > > > > > - prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT), > > > > - htons(ROCE_V2_UDP_DPORT)); > > > > + prepare_udp_hdr(skb, htons(qp->src_port), > > > > htons(ROCE_V2_UDP_DPORT)); > > > > > > > > prepare_ipv4_hdr(dst, skb, saddr->s_addr, daddr->s_addr, > > > > IPPROTO_UDP, > > > > av->grh.traffic_class, av->grh.hop_limit, df, xnet); @@ > > > > -440,8 +439,7 @@ static int prepare6(struct rxe_dev *rxe, struct > > > > rxe_pkt_info *pkt, > > > > if (!memcmp(saddr, daddr, sizeof(*daddr))) > > > > pkt->mask |= RXE_LOOPBACK_MASK; > > > > > > > > - prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT), > > > > - htons(ROCE_V2_UDP_DPORT)); > > > > + prepare_udp_hdr(skb, htons(qp->src_port), > > > > htons(ROCE_V2_UDP_DPORT)); > > > > > > > > prepare_ipv6_hdr(dst, skb, saddr, daddr, IPPROTO_UDP, > > > > av->grh.traffic_class, > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c > > > > b/drivers/infiniband/sw/rxe/rxe_qp.c > > > > index b9f7aa1..fa8bfa9 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe_qp.c > > > > +++ b/drivers/infiniband/sw/rxe/rxe_qp.c > > > > @@ -227,6 +227,8 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, > > > > struct rxe_qp *qp, > > > > return err; > > > > qp->sk->sk->sk_user_data = qp; > > > > > > > > + qp->src_port = RXE_ROCE_V2_SPORT; > > > > + > > > QP shouldn't be transmitting anything until it is in RTR state. So this > > initialization doesn't seem necessary. > > > > I think it's necessary for UD. The dest QP may not be set via the QP attr, so this > > initialization makes sure we're using an ephemeral port. > Yes, if rxe run's QP state machine, than right place to set is when qp_attr state is set to RTR regardless of QP type. > That makes is generic, unless you want to pick different src port for UD depending on the hash of src/dst ip during skb processing time. Ok, I'll set it where the QP state is set to RTR. I wasn't planning to pick a different src port for UD (during skb processing). > > > > > > > > > > qp->sq.max_wr = init->cap.max_send_wr; > > > > qp->sq.max_sge = init->cap.max_send_sge; > > > > qp->sq.max_inline = init->cap.max_inline_data; > > > > @@ -706,8 +708,12 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct > > > > ib_qp_attr *attr, int mask, > > > > if (mask & IB_QP_PATH_MIG_STATE) > > > > qp->attr.path_mig_state = attr->path_mig_state; > > > > > > > > - if (mask & IB_QP_DEST_QPN) > > > > + if (mask & IB_QP_DEST_QPN) { > > > > qp->attr.dest_qp_num = attr->dest_qp_num; > > > > + qp->src_port = RXE_ROCE_V2_SPORT + > > > > + (hash_64_generic(((u64)attr->dest_qp_num << 24) + > > > > + (u64)qp_num(qp), 14) & 0x3fff); > > > > + } > > > > > > > > if (mask & IB_QP_STATE) { > > > > qp->attr.qp_state = attr->qp_state; diff --git > > > > a/drivers/infiniband/sw/rxe/rxe_verbs.h > > > > b/drivers/infiniband/sw/rxe/rxe_verbs.h > > > > index af1470d..0d9200878 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe_verbs.h > > > > +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h > > > > @@ -248,6 +248,7 @@ struct rxe_qp { > > > > > > > > struct socket *sk; > > > > u32 dst_cookie; > > > > + u16 src_port; > > > > > > > > struct rxe_av pri_av; > > > > struct rxe_av alt_av; > > > > -- > > > > 2.7.4 > > > > > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe > > > > linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
