Hi, On 1 Apr 2014 at 15:59:07, Andrey Korolyov (andrey at xdel.ru<mailto:andrey at xdel.ru>) wrote: On 04/01/2014 03:51 PM, Robert Sander wrote: > On 01.04.2014 13:38, Karol Kozubal wrote: > >> I am curious to know what is the largest known ceph production deployment? > > I would assume it is the CERN installation. > > Have a look at the slides from Frankfurt Ceph Day: > > http://www.slideshare.net/Inktank_Ceph/scaling-ceph-at-cern > > Regards > Just curious, how CERN guys built the network topology to prevent possible cluster splits, because split in the middle will cause huge downtime even for a relatively short split time enough to mark half of those 1k OSDs as down by remaining MON majority. The mons are distributed around the data centre, across N switches. The OSDs are across a few switches ? actually, we could use CRUSH rules to replicate across switches but didn?t do so because of an (unconfirmed) fear that the uplinks would become a bottleneck. So a switch or routing outage scenario is clearly a point of failure where some PGs could become stale, but we?ve been lucky enough not to suffer from that yet. BTW, this 3PB cluster was built to test the scalability of Ceph's implementation, not because we have 3PB of data to store in Ceph today (most of the results of those tests are discussed in that presentation.). And we are currently partitioning this cluster down into a smaller production instance for Cinder and other instances for ongoing tests. BTW#2, I don?t think the CERN cluster is the largest. Isn?t DreamHost?s bigger? Cheers, Dan -- Dan van der Ster || Data & Storage Services || CERN IT Department -- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140401/eda50072/attachment.htm>
 |