On Thu, 5 Sep 2013, Bill Omer wrote: > Thats correct. We created 65k buckets, using two hex characters as the > naming convention, then stored the files in each container based on their > first two characters in the file name. The end result was 20-50 files per > bucket. Once all of the buckets were created and files were being loaded, > we still observed an increase in latency overtime. This might be going too far in the opposite direction. I would target 1000's of objects per bucket, not 10's. The radosgw has to validate bucket ACLs on requests. It caches them, but it probably can't cache 64K of them (not by default at least!). And even if it can, it will take a long long time for the cache to warm up. In any case, the end result is that there is probably an extra rados request going on on the backend for every request. Maybe try over ~1000 buckets and see how that goes? And give the cache a bit of time to warm up? sage > Is there a way to disable indexing? Or are there other settings you can > suggest to attempt to speed this process up? > > > On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote: > Just for clarification, distributing objects over lots of > buckets isn't helping improve small object performance? > > The degradation over time is similar to something I've seen in > the past, with higher numbers of seeks on the underlying OSD > device over time. Is it always (temporarily) resolved writing > to a new empty bucket? > > Mark > > On 09/04/2013 02:45 PM, Bill Omer wrote: > We've actually done the same thing, creating 65k buckets > and storing > 20-50 objects in each. No change really, not noticeable > anyway > > > On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell > <bstillwell@xxxxxxxxxxxxxxx <mailto:bstillwell@xxxxxxxxxxxxxxx>> > wrote: > > So far I haven't seen much of a change. It's still working > through > removing the bucket that reached 1.5 million objects though > (my > guess is that'll take a few more days), so I believe that > might have > something to do with it. > > Bryan > > > On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson > <mark.nelson@xxxxxxxxxxx <mailto:mark.nelson@xxxxxxxxxxx>> > wrote: > > Bryan, > > Good explanation. How's performance now that you've > spread the > load over multiple buckets? > > Mark > > On 09/04/2013 12:39 PM, Bryan Stillwell wrote: > > Bill, > > I've run into a similar issue with objects averaging > ~100KiB. The > explanation I received on IRC is that there are > scaling > issues if you're > uploading them all to the same bucket because the > index > isn't sharded. > The recommended solution is to spread the objects > out to > a lot of > buckets. However, that ran me into another issue > once I hit > 1000 > buckets which is a per user limit. I switched the > limit to > be unlimited > with this command: > > radosgw-admin user modify --uid=your_username > --max-buckets=0 > > Bryan > > > On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer > <bill.omer@xxxxxxxxx <mailto:bill.omer@xxxxxxxxx> > <mailto:bill.omer@xxxxxxxxx > <mailto:bill.omer@xxxxxxxxx>>> > wrote: > > I'm testing ceph for storing a very large > number of > small files. > I'm seeing some performance issues and would > like to > see if anyone > could offer any insight as to what I could do > to > correct this. > > Some numbers: > > Uploaded 184111 files, with an average file > size of > 5KB, using > 10 separate servers to upload the request using > Python > and the > cloudfiles module. I stopped uploading after > 53 > minutes, which > seems to average 5.7 files per second per node. > > > My storage cluster consists of 21 OSD's across > 7 > servers, with their > journals written to SSD drives. I've done a > default > installation, > using ceph-deploy with the dumpling release. > > I'm using statsd to monitor the performance, > and what's > interesting > is when I start with an empty bucket, > performance is > amazing, with > average response times of 20-50ms. However as > time > goes on, the > response times go in to the hundreds, and the > average > number of > uploads per second drops. > > I've installed radosgw on all 7 ceph servers. > I've > tested using a > load balancer to distribute the api calls, as > well as > pointing the > 10 worker servers to a single instance. I've > not seen > a real > different in performance with this either. > > > Each of the ceph servers are 16 core Xeon > 2.53GHz with > 72GB of ram, > OCZ Vertex4 SSD drives for the journals and > Seagate > Barracuda ES2 > drives for storage. > > > Any help would be greatly appreciated. > > > > _________________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx> > <mailto:ceph-users@xxxxxxxxxx.__com > <mailto:ceph-users@xxxxxxxxxxxxxx>> > > http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > > > -- > Photobucket <http://photobucket.com> > > *Bryan Stillwell* > SENIOR SYSTEM ADMINISTRATOR > > E: bstillwell@xxxxxxxxxxxxxxx > <mailto:bstillwell@xxxxxxxxxxxxxxx> > <mailto:bstillwell@__photobucket.com > <mailto:bstillwell@xxxxxxxxxxxxxxx>> > O: 303.228.5109 <tel:303.228.5109> > M: 970.310.6085 <tel:970.310.6085> > > Facebook <http://www.facebook.com/__photobucket > <http://www.facebook.com/photobucket>> Twitter > <http://twitter.com/__photobucket > <http://twitter.com/photobucket>> Photobucket > <http://photobucket.com/__images/photobucket > <http://photobucket.com/images/photobucket>> > > > > > _________________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx> > > http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > _________________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx> > > http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > > > -- > Photobucket <http://photobucket.com> > > *Bryan Stillwell* > > SENIOR SYSTEM ADMINISTRATOR > > E: bstillwell@xxxxxxxxxxxxxxx > <mailto:bstillwell@xxxxxxxxxxxxxxx> > O: 303.228.5109 <tel:303.228.5109> > M: 970.310.6085 <tel:970.310.6085> > > Facebook <http://www.facebook.com/photobucket> Twitter > <http://twitter.com/photobucket> Photobucket > <http://photobucket.com/images/photobucket> > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > >
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
