> Hello, > > I wanted to know if there are any max limitations on > > - Max number of Ceph data nodes > - Max number of OSDs per data node > - Global max on number of OSDs > - Any limitations on the size of each drive managed by OSD? > - Any limitation on number of client nodes? > - Any limitation on maximum number of RBD volumes that can be created? I don’t think there any *architectural* limits, but there can be *practical* limits. There are a lot of variables and everyone has a unique situation, but some thoughts: > Max number of Ceph data nodes May be limited at some extreme by networking. Don’t cheap out on your switches. > - Max number of OSDs per data node People have run at least 72. Consider RAM required for a given set of drives, and that a single host/chassis isn’t a big percentage of your cluster. Ie., don’t have a huge fault domain that will bite you later. For a production cluster at scale I would suggest at least 12 OSD nodes, but this depends on lots of variables. Conventional wisdom is 1GB RAM per 1TB of OSD; in practice for a large cluster I would favor somewhat more. A cluster with, say, 3 nodes of 72 OSDs each is going to be in bad way when one fails. > - Global max on number of OSDs A cluster with at lest 10800 has existed. https://indico.cern.ch/event/542464/contributions/2202295/attachments/1289543/1921810/cephday-dan.pdf https://indico.cern.ch/event/649159/contributions/2761965/attachments/1544385/2423339/hroussea-storage-at-CERN.pdf The larger a cluster becomes, the more careful attention must be paid to topology and tuning. > Also, any advise on using NVMes for OSD drives? They rock. Evaluate your servers carefully: * Some may route PCI through a multi-mode SAS/SATA HBA * Watch for PCI bridges or multiplexing * Pinning, minimize data over QPI links * Faster vs more cores can squeeze out more performance AMD Epyc single-socket systems may be very interesting for NVMe OSD nodes. > What is the known maximum cluster size that Ceph RBD has been deployed to? See above. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |