Greetings, Getting multiple infiniband interfaces on a node to work required more changes than the hints provided in Documentation/librdmacm.md. We have a few suggestions for additions to that page that might save others a lot of debugging time. 1. accept_local must be set to 1 as documented in in librdmacm.md. 2. The documentation says to set arp_ignore to the value 2. That only works as long as all the interfaces on clients and servers are in the same logical subnet, e.g. ib0 == 10.0.0.27, ib1 == 10.0.1.27, prefix=16. If you happened to create separate logical subnets, e.g. prefix=24, then the value 2 does not work, you have to use the value 1. 3. The documentation does not mention the rp_filter parameter, but it must be modified even if you have created a single logical subnet. The value of rp_filter cannot be 1, you must set it to either 0 or 2, with 2 being more secure. The default value on CentOS 7 is 1. We are using 2. We were surprised to find that the value of rp_filter would also get reset after a “systemctl restart network”, not just on reboots, so the setting you want must be persisted in a /etc/sysctl.d file if you want the value to stay across network restarts. The other parameters held their values across network restarts. With the above changes persisted in a new /etc/sysctl.d/95-daos.conf file, we have been able to successfully use multiple infiniband interfaces per node. Distribution is CentOS 7. Kernel is 3.10.0-1062.18.1.el7.x86_64 CentOS 7 sysctl defaults: net.ipv4.conf.default.accept_local = 0 net.ipv4.conf.default.arp_ignore = 0 net.ipv4.conf.default.arp_filter = 0 net.ipv4.conf.default.rp_filter = 1 If you need more information, let me know, Regards, Kevan
