> -----Original Message----- > From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma- > owner@xxxxxxxxxxxxxxx] On Behalf Of Jason Gunthorpe > Sent: Monday, June 18, 2018 9:51 PM > To: Vijay Immanuel <vijayi@xxxxxxxxxxxxxxxxx> > Cc: linux-rdma@xxxxxxxxxxxxxxx > Subject: Re: [PATCH] IB/rxe: vary the source udp port for receive scaling > > On Mon, Jun 18, 2018 at 06:47:15PM -0700, Vijay Immanuel wrote: > > Select the source udp port number for a QP based on a hash of the > > source and destination QPNs. This provides a better spread of traffic > > across NIC RX queues. > > > > Signed-off-by: Vijay Immanuel <vijayi@xxxxxxxxxxxxxxxxx> > > drivers/infiniband/sw/rxe/rxe_net.c | 6 ++---- > > drivers/infiniband/sw/rxe/rxe_qp.c | 8 +++++++- > > drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + > > 3 files changed, 10 insertions(+), 5 deletions(-) > > This seems like something that would be contrary to the protocol? > Proposed patch stores the source udp port during QP state transition, only once. So its fine. Protocol allows this. For connected QPs, it's not desired to change the src port number every packet, otherwise it can take different route in network leading to lower performance due to out-of-order delivery. Snippet from spec: The Source Port field in the UDP header of a RoCEv2 packet may be used by network devices as a component in the selection of a route among multiple possible alternative routes - see Section 17.9.4, "ECMP for RoCEv2," on page 21. For that reason, HCAs should use a fixed value across packets where ordering matters between them (e.g. packets of a connected QP). However this patch is not correct where it uses random udp source port number. ib_core needs to bind the source udp port number and store it into ib_qp. I have been holding that up, due to gid ref count series, so that I can use that logic for destination and source udp port and store the socket pointer in ib_gid_attr. Once that is available that that can be used by rxe and other drivers. However multiple src udp port number has benefits which can be a list per gid entry as mentioned in this patch. Per device would be simpler, but given that we are sharing the device among multiple namespaces in some cases, having per gid entry is simpler to start with. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
