I don't know what your disk performance in your OSD nodes are, but dual FDR would probably be more than enough that I wouldn't worry about doing a separate cluster network. The FDR card should have more capacity than your PCI bus anyway.
Since Ceph does not use RDMA or native IB verbs yet, you won't see a ton of improvement over your Ethernet implementation. Mellanox and others are working to have Ceph use AccelIO which will provide RDMA messages for Ceph, but there is still quite a bit of work to get it fully supported.
I haven't used the SX6036G, so I don't know what to expect or offer you in that regard, but it may determine if you use the Mellanox driver set or if you can use the OFED drivers (packaged with your distro). I personally like using the distro packages even if they are a little more out of date, but to support the gateway, you might need something newer.
We use connected mode on our cluster with MTU of 65520, but some people have better performance with datagram mode and MTU of 4096. I think if you are using RDMA, then datagram mode does perform better. You will need to test it in your environment and with the drivers you choose to use.
As far as Ceph, currently you would configure it the same as Ethernet with IPoIB. You just have to make sure you have a subnet manager up on the IB fabric so that you can bring up the IPoIB interfaces. For some reason CentOS7 doesn't bring up open-sm automatically, so I have to remember to start it when I reboot one of the boxes, luckily it can be clustered very easily.
I recommend searching through the mailing list archives, some tweaks for IB have been given for memory contention scenarios (this will be important if your OSDs don't have a lot of RAM since IB locks memory regions). Keep a look out for progress on XIO on the mailing list to see when native IB support will be in Ceph.
On Tue, Mar 17, 2015 at 12:13 PM, German Anders <ganders@xxxxxxxxxxxx> wrote:
Hi Robert,
How are you? Thanks a lot for the quick response. I would like to know if you could share some info on this. We have an existing Ceph cluster in production, with the following:
3 x MON servers with 10GbE ADPT DP (one port on the PUB network)
4 x OSD servers with 10GbE ADPT DP (one port on the PUB network and one port on the CLUS network)
We are using Juniper EX4500 at 10GbE
This cluster works fine but we need to improve performance on the network side, and we are going for other reason too, to implement Infiniband FDR 56Gb/s, I've spoke with Mellanox people and we are going to use (2) SX6036G gateway switches, so we can connect our existing ETH network (2 x 40G connections on each gateway), the gateways are going to be in an active/active configuration, also we are going to put one ADPT FDR DP on each node.
And then we are going to have two more switches SX6018F so use for CLUS Ceph network, and the PUB network of the Ceph nodes is going to be connected directly to the IB gateways switches.
What considerations do we need to have in order to use IB with Ceph?, also, do you have any procedure to configure IB with Ceph?, like dependencies to install on the hosts, etc.
Any help will really be appreciated.
Thanks in advance,German Anders
Storage System Engineer Leader
Despegar | IT Team
office +54 11 4894 3500 x3408
mobile +54 911 3493 7262
mail ganders@xxxxxxxxxxxx--- Original message ---
Asunto: Re: Ceph + Infiniband CLUS & PUB Network
De: Robert LeBlanc <robert@xxxxxxxxxxxxx>
Para: German Anders <ganders@xxxxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx <ceph-users@xxxxxxxxxxxxxx>
Fecha: Tuesday, 17/03/2015 15:06We have a test cluster with IB. We have both networks over IPoIB on the same IP subnet though (no cluster network configuration).On Tue, Mar 17, 2015 at 12:02 PM, German Anders <ganders@xxxxxxxxxxxx> wrote:Hi All,
Does anyone have Ceph implemented with Infiniband for Cluster and Public network?
Thanks in advance,German Anders
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
