Think "many objects, few pools". The number of pools do not scale well because of PG limitations. Keep a small number of pools with the proper number of PGs. See this tool for pool sizing:
By default, the developers have chosen a 4MB object size for the built-in clients. This is a sensible choice and will result in good performance for most workloads, but can depend on what type of operations you most frequently perform and how your client interacts with the cluster.
I will let some other folks chime in with firsthand experience, but I have worked with pools containing billions of objects and observed them functioning fine. A few issues I can foresee off the top of my head are potential underlying filesystem limits (should be okay) and keeping cluster operations to a minimum (resizing/deleting pools).
Since we're talking about scale, CERN's videos are interesting for examining the current challenges in Ceph at scale. (mostly hardware observations)
Yahoo chose a "super-cluster" architecture to work around former limitations with large clusters, but I do believe many of the findings CERN/Yahoo have uncovered have been addressed in recent versions of Ceph, or are being targeted by developers in upcoming versions.
On Sat, Dec 31, 2016 at 3:53 PM, Kent Borg <kentborg@xxxxxxxx> wrote:
More newbie questions about librados...
I am making design decisions now that I want to scale to really big sizes in the future, and so need to understand where size limits and performance bottlenecks come from. Ceph has a reputation for being able to scale to exabytes, but I don't see much on how one should sensibly get to such scales. Do I make big objects? Pools with lots of objects in them? Lots of pools? A pool that has a thousand objects of a megabyte each vs. a pool that has a million objects or a thousand bytes each: why should one take one approach and when should one take the other? How big can a pool get? Is a billion objects a lot, something that Ceph works to handle, or is it something Ceph thinks is no big deal? Is a trillion objects a lot? Is a million pools a lot? A billion pools? How many is "lots" for Ceph?
I plan to accumulate data indefinitely, I plan to add cluster capacity on a regular schedule, I want performance that doesn't degrade with size.
Where do things break down? What is the wrong way to scale Ceph?
Thanks,
-kb, the Kent who guesses putting all his data in a single xattr or single RADOS object would be the wrong way.
P.S. Happy New Year!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Brian Andrus
Cloud Systems Engineer
DreamHost, LLC
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
