On 1/20/2019 10:53 PM, Alex Rosenbaum wrote:
thanks for taking a look and commenting.
On Sun, Jan 20, 2019 at 7:42 PM Tom Talpey <tom@xxxxxxxxxx> wrote:
On 1/15/2019 3:53 AM, Alex Rosenbaum wrote:
RoCE v2.0 UDP source port is used to provide per-conversation
entropy for Network Routers (ECMP), load balancers and 802.3ad Link
Aggregation Switching that are not aware of RoCE headers.
We will use the following hash functions definition to calculate the
RoCE UDP src_port according to the service type (QP Type) and Service ID
(SID), or according to the source and destination QPN values where CM
services are not used.
Won't this result in the source port being the same for all connections
to a given remote service? In other words, all NFS/RDMA connections to a
given server will have the same src_port, and therefore will all stack
up in the same router queues? Similarly for SMB, iSER, etc.
It's true that the Remote Service ID (mapped to a DstPort) is the same
for all connections.
But the local bounded port (SrcPort) of CM, for each connection will
be different.
Each RC QP, will have its separate CM 'initiator' handle, which will
have a seperate local bounded src port.
You can see in the below hash calc for CM based traffic, that the
SrcPort (local bound CM port) is used in the hash, so that multiple
connection to the same remote will result is scattered UDP.src_prot
values.
Extract the following fields from the CM IP REQ:
CM_REQ.ServiceID.DstPort [2 Bytes]
CM_REQ.PrivateData.SrcPort [2 Bytes]
RoCE.UDP.src_port = (DstPort[0..1] XOR SrcPort[0..1]) OR 0xC000
Ah, ok I did indeed misread the fields, and as long as the CM's
SrcPort assignment has entropy, this approach may be ok.
But, my concern then, is that the "hash" appears to me quite lossy.
An XOR of two ports is not going to lead to a lot of entropy,
especially since for a given service, the DstPort may be common
among many connections. And, OR'ing the result with 0xC000 to
force into the dynamic port range means that the high two bits
are constant. I'd guess that real entropy is down to ~10 bits.
Can you propose a hash that takes two full-16-bit values and
produces a true 14-bit result? Mathematically, that would a far
better scheme than the 16x16 XOR proposed.
Tom.
