Hello, I've resent the mail with correct subject. I've noticed that RDMA doesn't work on IPv6 between 2 NIC on the same host. I've tested with rping and ibv_rc_pingpong. Below you can find the commands and the behavior for each app. rping test: > rping -s -a 2001:db8:153:1::10 -p 60009 -V -d // waits for a connection > rping -c -a 2001:db8:153:1::10 -I 2001:db8:171:2::20 -p 60009 -V -d // client waits indefinitely to connect to server ibv_rc_pingpong test: > ibv_rc_pingpong -d mlx5_2 -g 5 // waits for a connection > ibv_rc_pingpong -d mlx5_3 -g 5 <IPv4 address> //client throws timeout error: local address: LID 0x0000, QPN 0x000150, PSN 0x23c9c7, GID 2001:db8:171:2::20 remote address: LID 0x0000, QPN 0x00012e, PSN 0x96791a, GID 2001:db8:153:1::10 Failed status transport retry counter exceeded (12) for wr_id 2 parse WC failed 1 What I have noticed with tcpdump on mlx5_3 is that the packets have Source MAC equal to Destination MAC, which is wrong! Also, I want to mention that both interfaces have also IPv4 addresses and if I'm using IPv4 everything works perfect! Below you can find my setup: Ubuntu 22.0.4.5 (6.8.0-49-generic #49~22.04.1-Ubuntu) with 2 NICs connected via a switch. Network interfaces: enp153s0np0: ... inet6 fe80::a288:c2ff:fe7d:a8ca prefixlen 64 scopeid 0x20<link> inet6 2001:db8:153:1::10 prefixlen 64 scopeid 0x0<global> ether a0:88:c2:7d:a8:ca txqueuelen 1000 (Ethernet) and enp171s0np0: ... inet6 fe80::a288:c2ff:fe7d:a88a prefixlen 64 scopeid 0x20<link> inet6 2001:db8:171:2::20 prefixlen 64 scopeid 0x0<global> ether a0:88:c2:7d:a8:8a txqueuelen 1000 (Ethernet) Here is the output of the ibv_devinfo: hca_id: mlx5_2 transport: InfiniBand (0) fw_ver: 28.39.1002 node_guid: a088:c203:007d:a8ca sys_image_guid: a088:c203:007d:a8ca vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_3 transport: InfiniBand (0) fw_ver: 28.39.1002 node_guid: a088:c203:007d:a88a sys_image_guid: a088:c203:007d:a88a vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet Thank you, Andrei
