On 2018/6/19 11:07, Parav Pandit wrote:
-----Original Message-----
From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma-
owner@xxxxxxxxxxxxxxx] On Behalf Of Jason Gunthorpe
Sent: Monday, June 18, 2018 9:51 PM
To: Vijay Immanuel <vijayi@xxxxxxxxxxxxxxxxx>
Cc: linux-rdma@xxxxxxxxxxxxxxx
Subject: Re: [PATCH] IB/rxe: vary the source udp port for receive scaling
On Mon, Jun 18, 2018 at 06:47:15PM -0700, Vijay Immanuel wrote:
Select the source udp port number for a QP based on a hash of the
source and destination QPNs. This provides a better spread of traffic
across NIC RX queues.
Signed-off-by: Vijay Immanuel <vijayi@xxxxxxxxxxxxxxxxx>
drivers/infiniband/sw/rxe/rxe_net.c | 6 ++----
drivers/infiniband/sw/rxe/rxe_qp.c | 8 +++++++-
drivers/infiniband/sw/rxe/rxe_verbs.h | 1 +
3 files changed, 10 insertions(+), 5 deletions(-)
This seems like something that would be contrary to the protocol?
Proposed patch stores the source udp port during QP state transition, only once.
So its fine. Protocol allows this.
For connected QPs, it's not desired to change the src port number every packet, otherwise it can take different route in network leading to lower performance due to out-of-order delivery.
Agree with you since RoCE is routable. The varying src port will take
different route in network to cause out-of-order delivery.
Snippet from spec:
The Source Port field in the UDP header of a RoCEv2 packet may be used by network devices as a component in the selection of a route among multiple possible alternative routes - see Section 17.9.4, "ECMP for RoCEv2," on page 21. For that reason, HCAs should use a fixed value across packets where ordering matters between them (e.g. packets of a connected QP).
However this patch is not correct where it uses random udp source port number.
ib_core needs to bind the source udp port number and store it into ib_qp.
I have been holding that up, due to gid ref count series, so that I can use that logic for destination and source udp port and store the socket pointer in ib_gid_attr.
Once that is available that that can be used by rxe and other drivers.
However multiple src udp port number has benefits which can be a list per gid entry as mentioned in this patch.
Per device would be simpler, but given that we are sharing the device among multiple namespaces in some cases, having per gid entry is simpler to start with.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
