We are close to being given approval to deploy a 3.5PB Ceph cluster that will be distributed over every major capital in Australia. The config will be dual sites in each city that will be coupled as HA pairs - 12 sites in total. The vast majority of CRUSH rules will place data either locally to the individual site, or replicated to the other HA site in that city. However there are future use cases where I think we could use EC to distribute data wider or have some replication that puts small data sets across multiple cities. All of this will be tied together with a dedicated private IP network. The concern I have is around the placement of mons. In the current design there would be two monitors in each site, running separate to the OSDs as part of some hosts acting as RBD to iSCSI/NFS gateways. There will also be a "tiebreaker" mon placed on a separate host which will house some management infrastructure for the whole platform. Obviously a concern is latency - the east coast to west coast latency is around 50ms, and on the east coast it is 12ms between Sydney and the other two sites, and 24ms Melbourne to Brisbane. Most of the data traffic will remain local but if we create a single national cluster then how much of an impact will it be having all the mons needing to keep in sync, as well as monitor and communicate with all OSDs (in the end goal design there will be some 2300+ OSDs). The other options I am considering: - split into east and west coast clusters, most of the cross city need is in the east coast, any data moves between clusters can be done with snap replication - city based clusters (tightest latency) but loose the multi-DC EC option, do cross city replication using snapshots Just want to get a feel for what I need to consider when we start building at this scale. Cheers, Adrian Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |