Re: EC cluster design considerations

[Paul Evans <paul@xxxxxxxxxxxx>]

On Jul 4, 2015, at 2:44 PM, Adrien Gillard <gillard.adrien@xxxxxxxxx> wrote:

Lastly, regarding Cluster Throughput:  EC seems to require a bit more CPU and memory than straight replication, which begs the question of how much RAM and CPU are you putting into the chassis?  With proper amounts, you should be able to hit your throughput targets,.  

Yes, I have read about that, I was thinking 64 GB of RAM  (maybe overkill, even with the 1GB of RAM per TB ? but I would rather have an optimal RAM configuration in terms of DIMM / channels / CPU) and 2x8 Intel cores per host (around 2Ghz per core). As the cluster will be used for backups, the goal is not to be limited by the storage backend during the backup window overnight. I do not expect much load during daytime.

64G is “OK” provided you tune the system well and DON”T add extra services onto your OSD nodes.  If you’ll also have 3 of them acting as MONs, more memory is advised (probably 96-128G). 

 

 At the moment I am planning to have a smaller dedicated node for the master monitor ( ~ 8 cores, 32G RAM, SSD) and virtual machines for MON 2 and 3 (with enough resources and virtual disk on SSD)

It would be good to have others comment on the practicality of this design, as I don’t believe there is a benefit to having a single MON that is ‘better' than the other two. My reasoning comes from a limited understanding of the Paxos implementation within Ceph, which suggests that a majority of MONs must be available at all times (i.e. - 2 of the 3), and that MON activities will be processed according to the speed of the slowest quorum member.  If two of the MONs are running as VMs on OSD hosts, and you have a write-heavy workload, I can foresee some interesting resource contention issues that might sometimes destabilize your entire cluster.   YMMV. 

- Paul

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com