Hi, I'm currently trying to set up a brand new home cluster : - 5 nodes, with each : - 1 HCA Mellanox ConnectX-2 - 1 GB Ethernet (Proxmox 5.1 Network Admin) - 1 CX4 to CX4 cable All together connected to a SDR Flextronics IB Switch. This setup should back a Ceph Luminous (V12.2.2 included in proxmox V5.1) On all nodes, I did: - apt-get infiniband-diags - modprobe mlx4_ib - modprobe ib_ipoib - modprobe ib_umad - ifconfig ib0 IP/MASK On two nodes (tried previously on a single on, same issue), i installed opensm ( The switch doesn't have SM included) : apt-get install opensm /etc/init.d/opensm stop /etc/init.d/opensm start (Necessary to let the daemon create the logfiles) I tailed the logfile and got a "Active&Running" Setup, with "SUBNET UP" Every node is OK regardless to IB Setup : - All ib0 are UP, using ibstat - ibhosts and ibswitches seem to be OK On a node : ibping -S On every other node : ibping -G GID_Of_Previous_Server_Port I got a nice pong reply on every node. Should be happy, but... But i never went further.. Tried to ping each other. No way to get into this (mostly probably) simple issue... Any hint to achieve this task ?? Thanks for all Best regards _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |