Re: ibv_rc_pingpong failes on softroce device using GID[0]

Yanjun Zhu <yanjun.zhu@xxxxxxxxx> · Fri, 4 Feb 2022 23:59:37 +0800

在 2022/2/3 13:28, hui wang 写道:
Hi All
I am using ubuntu 21.10. I used the following command to create a
softroce device rxe0:
sudo rdma link add rxe0 type rxe netdev enp0s3

rxe0's GID[0] is not an real IP address assigned to my enp0s3. It
seems to be something softroce generated from the node_guid (seems to
be generated from mac address of enp0s3).
Is this expected behavior? How to use this GID given that it is not
reachable because it is not a real IP address?

I used ibv_rc_pingpong -d rxe0 -g 0 to setup a server with this GID,
the server failed with "Failed to modify QP to RTR" when client tries

please check this link:

https://www.spinics.net/lists/linux-rdma/msg108317.html

Zhu Yanjun

to talk with it with client's GID[0] using command "./ibv_rc_pingpong
-d rxe0 -g 0 10.0.0.233". Is this because the GID[0] is not a real IP?

ibv_rc_pingpong works if i use any other GID, (eg. GID[1] - GID[5])

Here is the output of ibv_devinfo and ip addr. you can see GID[0]
fe80::a00:27ff:fe2e:f7a5 is not a real ip address assigned to enp0s3
(where rxe0 was created).

huiwang@huiwang-VirtualBox:~/rdma-core$ ibv_devinfo -v
hca_id: rxe0
         transport:                      InfiniBand (0)
         fw_ver:                         0.0.0
         node_guid:                      0a00:27ff:fe2e:f7a5
         sys_image_guid:                 0a00:27ff:fe2e:f7a5
....
                         GID[  0]:
fe80::a00:27ff:fe2e:f7a5, RoCE v2
                         GID[  1]:
fe80::9bd3:885:53df:9fc4, RoCE v2
                         GID[  2]:               ::ffff:10.0.0.151, RoCE v2
                         GID[  3]:
2601:647:5380:2210:19df:d855:54a9:cc9f, RoCE v2
                         GID[  4]:
2601:647:5380:2210:e3f3:7e57:504b:7e5c, RoCE v2
                         GID[  5]:
2601:647:5380:2210::3be1, RoCE v2

huiwang@huiwang-VirtualBox:~/rdma-core/build/bin$ ip addr
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
state UP group default qlen 1000
     link/ether 08:00:27:2e:f7:a5 brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.151/64 brd 10.0.0.255 scope global dynamic noprefixroute enp0s3
        valid_lft 143273sec preferred_lft 143273sec
     inet6 2601:647:5380:2210::3be1/128 scope global dynamic noprefixroute
        valid_lft 575276sec preferred_lft 575276sec
     inet6 2601:647:5380:2210:e3f3:7e57:504b:7e5c/64 scope global
temporary dynamic
        valid_lft 255135sec preferred_lft 56812sec
     inet6 2601:647:5380:2210:19df:d855:54a9:cc9f/64 scope global
dynamic mngtmpaddr noprefixroute
        valid_lft 255135sec preferred_lft 255135sec
     inet6 fe80::9bd3:885:53df:9fc4/64 scope link noprefixroute
        valid_lft forever preferred_lft forever

Thanks,
Hui