Hi Stephen, If you only intention is to force to outgoing port and loopback externally, I can suggest you different route table configuration without NAT. Let me know. Parav > -----Original Message----- > From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma- > owner@xxxxxxxxxxxxxxx] On Behalf Of Stephen Bates > Sent: Monday, September 18, 2017 1:18 PM > To: linux-rdma@xxxxxxxxxxxxxxx > Cc: Logan Gunthorpe <logang@xxxxxxxxxxxx>; Sagi Grimberg > <sagi@xxxxxxxxxxx>; Max Gurtovoy <maxg@xxxxxxxxxxxx> > Subject: [linux-rdma and rdma-core]: Unable to perform rdma_connect in > loopbacked configuration > > Hi All > > I am seeing an issue that I think is a problem with rdma_cm and wanted to > report it here to see if anyone has any advice. Basically, I have two HCAs in a > single server connected via a network cable. I can perform ping, iperf and other > IP related applications and I see traffic flow out one NIC, over the cable and in > the other NIC.. However, any rdma_cm related applications fail at the > rdma_connect step. [BTW I have confirmed that things work fine in a more > traditional setup using two servers.] > > The Details > > 1. 4.12.3 stable kernel. > 2. rdma-core v14. > 3. Mellanox CX5 100G HCAs configured for Ethernet (RoCE) mode. > 4. Intel x86_64 CPU. > > Using a NAT approach discussed in [1] I can setup IPv4 addresses on both HCAs > such that I avoid a local loopback (the addresses I use are a little different to the > ones in that reference but the approach is identical). This allows ping, iperf and > other IP based applications to work just fine. For example: > > <server> > iperf –B 172.18.1.1 > </server> > <client> > iperf 172.18.11.1 > </client> > > works great and I can use packet counters to confirm the traffic is hitting the > network cable. > > However, if I try: > > <server> > rping –s –a 172.18.1.1 -vVd > </server> > <client> > rping –c –a 172.18.11.1 –vVd > </client> > > I see the following: > > <server> > created cm_id 0xceded03170 > rdma_bind_addr successful > rdma_listen > </server> > <client> > created cm_id 0x138702d110 > cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x138702d110 > (parent) cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id > 0x138702d110 (parent) rdma_resolve_addr - rdma_resolve_route successful > created pd 0x138702cb80 created channel 0x138702cba0 created cq > 0x138702cbc0 created qp 0x138702faf8 rping_setup_buffers called on cb > 0x13870253c0 allocated & registered buffers... > cq_thread started. > cma_event type RDMA_CM_EVENT_UNREACHABLE cma_id 0x138702d110 > (parent) cma event RDMA_CM_EVENT_UNREACHABLE, error -110 wait for > CONNECTED state 4 connect error -1 </client> > > I’ve tried using configfs to switch the preferred RoCE mode but that had no > effect. I’d appreciate any ideas or input from anyone who might have got this > working on their systems. I know there are other ways to solve this (e.g. > (para)virtualization of the client) but I’d like to get this approach up and running > if I can). BTW as an extra piece of input I also tried using in-kernel rdma_cm > (using NVMe over Fabrics) and got a similar error message… > > Cheers > > Stephen > > [1] > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fserverf > ault.com%2Fquestions%2F127636%2Fforce-local-ip-traffic-to-an-external- > interface&data=02%7C01%7Cparav%40mellanox.com%7Ce2f8f804df59419cc25 > a08d4fec1b338%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C63641 > 3555309452199&sdata=E2MQTCbSTHiUxmIAVZITBCPgKTEWBTY2%2BWVqHuSJ > KlY%3D&reserved=0 > > > {.n + +% lzwm b 맲 r zX ݙ ܨ} Ơz &j:+v zZ+ +zf h ~ i z w ? > & )ߢf ��.n��������+%������w��{.n�����{���fk��ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
