Thanks for your answers, Mark. The Nvidia documentation[1] states the following: "RoCE has two addressing modes: MAC based GIDs, and IP address based GIDs." Do you know which address family (AF_*) should be used to establish connection with MAC based GIDs? Is there an example or some documentation somewhere? I can fix ucmatose myself, but I have no idea what's missing exactly. [1]: https://docs.nvidia.com/networking/pages/viewpage.action?pageId=15046549 On Tue, Mar 8, 2022 at 1:43 PM Mark Zhang <markzhang@xxxxxxxxxx> wrote: > > On 3/8/2022 6:54 PM, Sylvain Didelot wrote: > > External email: Use caution opening links or attachments > > > > > > Does it mean that RoCE always requires the network interface to have > > an IP address as it cannot use the GID to establish connections? > > > > I think it's just a ucmatose limitation.. > > > On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@xxxxxxxxxx> wrote: > >> > >> On 3/7/2022 9:57 PM, Sylvain Didelot wrote: > >>> External email: Use caution opening links or attachments > >>> > >>> > >>> Hi, > >>> > >>> I have configured one of my Mellanox network adapters for RoCE: > >>> --- > >>> CA 'roceP1p1s0f1' > >>> CA type: MT4123 > >>> Number of ports: 1 > >>> Firmware version: 20.32.1010 > >>> Hardware version: 0 > >>> Node GUID: 0xb8cef603002d1707 > >>> System image GUID: 0xb8cef603002d1706 > >>> Port 1: > >>> State: Active > >>> Physical state: LinkUp > >>> Rate: 100 > >>> Base lid: 0 > >>> LMC: 0 > >>> SM lid: 0 > >>> Capability mask: 0x00010000 > >>> Port GUID: 0xbacef6fffe2d1707 > >>> Link layer: Ethernet > >>> --- > >>> > >>> The Infiniband stack was installed from the official Ubuntu repository > >>> (20.04.4 LTS): > >>> --- > >>> $ apt search ibverbs > >>> Sorting... Done > >>> Full Text Search... Done > >>> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed] > >>> User space provider drivers for libibverbs > >>> > >>> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed] > >>> Examples for the libibverbs library > >>> > >>> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed] > >>> Development files for the libibverbs library > >>> > >>> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed] > >>> Library for direct userspace use of RDMA (InfiniBand/iWARP) > >>> > >>> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed] > >>> Development files for the librdmacm library > >>> > >>> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed] > >>> Library for managing RDMA connections > >>> > >>> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed] > >>> Examples for the librdmacm library > >>> --- > >>> > >>> When I start the ucmatose server with the address format "gid", the > >>> tool fails binding with the error "No such device" > >>> > >>> Here is an example of the output: > >>> --- > >>> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0 > >>> fe80:0000:0000:0000:bace:f6ff:fe2d:1707 > >>> > >>> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid > >>> cmatose: starting server > >>> cmatose: bind address failed: No such device > >>> test complete > >>> return status -1 > >>> --- > >>> > >>> Does rdmacm support connection establishment using GID with RoCE? Or > >>> Is it a known limitation for RoCE? > >>> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly. > >>> > >>> Thanks for your help and your feedback. > >>> > >>> Sylvain Didelot > >> > >> I think ucmatose doesn't support RoCE when using "-f gid", as in this > >> case ai_family is set to AF_IB. >
