On 3/8/2022 6:54 PM, Sylvain Didelot wrote:
External email: Use caution opening links or attachments
Does it mean that RoCE always requires the network interface to have
an IP address as it cannot use the GID to establish connections?
I think it's just a ucmatose limitation..
On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@xxxxxxxxxx> wrote:
On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
External email: Use caution opening links or attachments
Hi,
I have configured one of my Mellanox network adapters for RoCE:
---
CA 'roceP1p1s0f1'
CA type: MT4123
Number of ports: 1
Firmware version: 20.32.1010
Hardware version: 0
Node GUID: 0xb8cef603002d1707
System image GUID: 0xb8cef603002d1706
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xbacef6fffe2d1707
Link layer: Ethernet
---
The Infiniband stack was installed from the official Ubuntu repository
(20.04.4 LTS):
---
$ apt search ibverbs
Sorting... Done
Full Text Search... Done
ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
User space provider drivers for libibverbs
ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
Examples for the libibverbs library
libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
Development files for the libibverbs library
libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
Library for direct userspace use of RDMA (InfiniBand/iWARP)
librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
Development files for the librdmacm library
librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
Library for managing RDMA connections
rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
Examples for the librdmacm library
---
When I start the ucmatose server with the address format "gid", the
tool fails binding with the error "No such device"
Here is an example of the output:
---
$ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
fe80:0000:0000:0000:bace:f6ff:fe2d:1707
$ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
cmatose: starting server
cmatose: bind address failed: No such device
test complete
return status -1
---
Does rdmacm support connection establishment using GID with RoCE? Or
Is it a known limitation for RoCE?
FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
Thanks for your help and your feedback.
Sylvain Didelot
I think ucmatose doesn't support RoCE when using "-f gid", as in this
case ai_family is set to AF_IB.
