On 12/17/2015 06:29 AM, Ben Hines wrote: > > > On Wed, Dec 16, 2015 at 11:05 AM, Florian Haas <florian@xxxxxxxxxxx > <mailto:florian@xxxxxxxxxxx>> wrote: > > Hi Ben & everyone, > > > Ben, you wrote elsewhere > (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-August/003955.html) > that you found approx. 900k objects to be the threshold where index > sharding becomes necessary. Have you found that to be a reasonable > rule of thumb, as in "try 1-2 shards per million objects in your most > populous bucket"? Also, do you reckon that beyond that, more shards > make things worse? > > > > Oh, and to answer this part. I didn't do that much experimentation > unfortunately. I actually am using about 24 index shards per bucket > currently and we delete each bucket once it hits about a million > objects. (it's just a throwaway cache for us) Seems ok, so i stopped > tweaking. > I have a use case where I need to store 350 Million objects in a single bucket. I tested with 4096 shards and that works. Creating the bucket takes a few seconds though. This setup is for archiving purposes, so data is written and not read that much afterwards. > Also, i think i have a pretty slow cluster as far as write speed is > concerned, since we do not have SSD Journals. With SSD journals i > imagine the index write speed is significantly improved, but i am not > sure how much. A faster cluster could probably handle bigger indexes. > > -Ben > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |