On Mon, Dec 21, 2015 at 12:36 PM, Wido den Hollander <wido@xxxxxxxx> wrote: > > > On 21-12-15 10:34, Florian Haas wrote: >> On Mon, Dec 21, 2015 at 10:20 AM, Wido den Hollander <wido@xxxxxxxx> wrote: >>>>>> Oh, and to answer this part. I didn't do that much experimentation >>>>>> unfortunately. I actually am using about 24 index shards per bucket >>>>>> currently and we delete each bucket once it hits about a million >>>>>> objects. (it's just a throwaway cache for us) Seems ok, so i stopped >>>>>> tweaking. >>>>>> >>>>> >>>>> I have a use case where I need to store 350 Million objects in a single >>>>> bucket. >>>> >>>> How many OSDs are in that cluster? >>>> >>> >>> 1800 and it will grow towards 2500 in Q1 2016. >>> >>>>> I tested with 4096 shards and that works. Creating the bucket takes a >>>>> few seconds though. >>>> >>>> Does "that works" mean that you have actually uploaded 350M objects into >>>> that one bucket? >>>> >>> >>> No, still in progress. The bucket functions, that is what I meant. >> >> Yep. What's your OSD LevelDB size (overall size of the OSD omap directory)? >> > > I'll take a look at that. This cluster is remotely where I can't access > it right now. > >> Do you happen to have rest-bench results created when the cluster was >> empty, and if so, what does rest-bench look like after you inject, >> say, 100M objects? >> > > Same story, I'll do that when I am there again. > >>>> If so, can you give me a feel for your typical object size? >>>> >>> >>> It varies. It is a archiving solution and I'm not in control there. >> >> Is there a "typical" size at least by order of magnitude? Kilobytes? >> Tens, hundreds of KBs? MBs? >> > > MBs mainly. Ranging from a few MB to tens or maybe hundreds. > >>>> Also, what's the performance drop you saw in bucket listing, vs. having >>>> fewer shards or no sharding at all? >>>> >>> >>> There is a drop in listing performance, didn't completely measure it, >>> but I think that with 4k shards the listing was a few seconds. >> >> Yeah, that sounds about expected. This would hurt if for some reason >> your use case involved having to list the bucket before inserting an >> object. >> > > Indeed and our case doesn't. Keep in mind though that AWS S3 also > recommends you not to list that often. > >>> In this use-case we are not going to list the bucket, ever. >> >> Never say never. :) >> > > Ok, ok. I won't :) > > Wido Thanks Wido, looking forward to your performance findings when you share them. :) Cheers, Florian _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |