Like Alex said network is not an issue now. For example I got a 6 node cluster with mix of sas and ssd disks running inside cassandra clusters with heavy load and also mysql clusters and Im getting less than 1ms of IO latency, on the network part i have infiniband with FDR configured and also cluster and public network running on the same ib network, no separation of clus network and runs so far so good! The ib network is shared by a total of 128 host plus the 6 node and 3 mon ceph cluster.
We are Planning now a all-nvme ceph cluster with ib. It would be nice that ceph works on the near future with the option of rdma integrated anyone know if this is on the roadmap? I know that there's a "possible" config of that but it's not production ready and needs a lot of tuning and configuration.
Best
On Fri, Mar 24, 2017 at 11:04 Alejandro Comisario <alejandro@xxxxxxxxxxx> wrote:
thanks for the recommendations so far.any one with more experiences and thoughts?best_______________________________________________On Mar 23, 2017 16:36, "Maxime Guyot" <Maxime.Guyot@xxxxxxxxx> wrote:Hi Alexandro,
As I understand you are planning NVMe for Journal for SATA HDD and collocated journal for SATA SSD?
Option 1:
- 24x SATA SSDs per server, will have a bottleneck with the storage bus/controller. Also, I would consider the network capacity 24xSSDs will deliver more performance than 24xHDD with journal, but you have the same network capacity on both types of nodes.
- This option is a little easier to implement: just move nodes in different CRUSHmap root
- Failure of a server (assuming size = 3) will impact all PGs
Option 2:
- You may have noisy neighbors effect between HDDs and SSDs, if HDDs are able to saturate your NICs or storage controller. So be mindful of this with the hardware design
- To configure the CRUSHmap for this you need to split each server in 2, I usually use “server1-hdd” and “server1-ssd” and map the right OSD in the right bucket, so a little extra work here but you can easily fix a “crush location hook” script for it (see example http://www.root314.com/2017/01/15/Ceph-storage-tiers/)
- In case of a server failure recovery will be faster than option 1 and will impact less PGs
Some general notes:
- SSD pools perform better with higher frequency CPUs
- the 1GB of RAM per TB is a little outdated, the current consensus for HDD OSDs is around 2GB/OSD (see https://www.redhat.com/cms/managed-files/st-rhcs-config-guide-technology-detail-inc0387897-201604-en.pdf)
- Network wise, if the SSD OSDs are rated for 500MB/s and use collocated journal you could generate up to 250MB/s of traffic per SSD OSD (24Gbps for 12x or 48Gbps for 24x) therefore I would consider doing 4x10G and consolidate both client and cluster network on that
Cheers,
Maxime
On 23/03/17 18:55, "ceph-users on behalf of Alejandro Comisario" <ceph-users-bounces@xxxxxxxxxxxxxx on behalf of alejandro@xxxxxxxxxxx> wrote:
Hi everyone!
I have to install a ceph cluster (6 nodes) with two "flavors" of
disks, 3 servers with SSD and 3 servers with SATA.
Y will purchase 24 disks servers (the ones with sata with NVE SSD for
the SATA journal)
Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the
OS, and 1.3GB of ram per storage TB.
The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb
for cluster network.
My doubts resides, ar want to ask the community about experiences and
pains and gains of choosing between.
Option 1
3 x servers just for SSD
3 x servers jsut for SATA
Option 2
6 x servers with 12 SSD and 12 SATA each
Regarding crushmap configuration and rules everything is clear to make
sure that two pools (poolSSD and poolSATA) uses the right disks.
But, what about performance, maintenance, architecture scalability, etc ?
thank you very much !
--
Alejandrito
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
German Anders
Storage Engineer Leader
Despegar | IT Team
office +54 11 4894 3500 x3408
mobile +54 911 3493 7262
mail ganders@xxxxxxxxxxxx
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
