On Thu, May 18, 2017 at 12:51 PM, Robert LeBlanc <robert@xxxxxxxxxxxxx> wrote: > On Sat, May 13, 2017 at 11:55 AM, Or Gerlitz <gerlitz.or@xxxxxxxxx> wrote: >> On Sat, May 13, 2017 at 12:02 AM, Robert LeBlanc <robert@xxxxxxxxxxxxx> wrote: >> [...] >>> If we put a default gateway for IPv6 in the 'default' table, then iSER >>> is able to make a connection, but we can only use one port. It looks >>> as if iSER is not following the rules in the default routing table to >>> find the appropriate default gateway in a different table. >> >> can you just try to make rdma connection with the rping utility? do >> you see the same problem/s? > > Using mlx4 cards with IB/RoCE, I am seeing the same connection problem > with rping. There seems to be a bug that prevents rping from working > with RoCE on mlx5. > > I also found a thread between you and Mike Christie > https://groups.google.com/forum/#!topic/open-iscsi/IywifztV7Xs. It > doesn't say that things were all worked out about binding to dev/ip > address. Should iSER be able to bind to ip address (my preference), or > by device name (workable) to provide multipathing? Should this binding > be able to work with additional tables for finding the proper gateway? > We are intending to route all RoCE traffic over L3 boundaries as it > greatly simplifies the network configuration. > > Here is the test that I ran to show the problem (it is also in the > "rdma_cm unable to find route on sub-table" thread): Test #1 ------- Using ConnectX-3 and Infiniband with two hosts back to back. Server ------ # systemctl start opensm # ip -6 addr add dev ib0 fd00::13/64 # ip link set ib0 up # rping -s -a ::0 server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10 Client ------ # ip -6 addr add dev ib0 fd00::14/64 # ip link set ib0 up # ping6 fd00::13 PING fd00::13(fd00::13) 56 data bytes 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.927 ms 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.134 ms ^C --- fd00::13 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 0.134/0.530/0.927/0.397 ms # rping -c -a fd00::13 ^C Results #1 ---------- Tests complete successfully Test #2 ------- Built upon the configuration of Test #1 Server ------ # rping -s -a ::0 Client ------ # echo "100 r0" >> /etc/iproute2/rt_tables # ip -6 route del fd00::/64 dev ib0 # ip -6 rule add from fd00::14 table r0 # ip -6 route add fd00::/64 dev ib0 table r0 # ping6 fd00::13 PING fd00::13(fd00::13) 56 data bytes 64 bytes from fd00::13: icmp_seq=1 ttl=64 time=0.158 ms 64 bytes from fd00::13: icmp_seq=2 ttl=64 time=0.136 ms ^C --- fd00::13 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1020ms rtt min/avg/max/mdev = 0.136/0.147/0.158/0.011 ms # rping -c -a fd00::13 rdma_resolve_addr: Network is unreachable Results #2 ---------- rping is unable to find a route to use ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
