Awesome, thanks a lot ! I'll try it. Paul Emmerich <paul.emmerich@xxxxxxxx> 于2020年5月18日周一 下午8:53写道： > > On Mon, May 18, 2020 at 1:52 PM Zhenshi Zhou <deaderzzs@xxxxxxxxx> wrote: > >> >> 50KB, and much video files around 30MB. The amount of the files is more >> than >> 1 million. Maybe I can find a way to seperate the files in more buckets so >> that >> there is no more than 1M objects in each bucket. But how about the small >> files >> around 50KB. Does rgw serve well on small files? >> > > 1 million files is usually the point where you first need to start > thinking about some optimizations, but that's mostly just making sure that > the index is on SSD and it'll happily work up to ~10 million files. > Then you might need to start thinking about the index being on *good* SSDs > (and/or on many SSDs/DB devices). > > It starts the get interesting if you need to go beyond 100 million files, > that's the point where you need to start tuning shard sizes and the types > of index queries that you send... > > I've found that a few hundred million objects per bucket are no problem if > you run with large shard sizes (500k - 1 million); however, there are some > index-queries that can be really expensive like filtering on prefixes in > some pathological cases... > > Small files: sure, works well, but can be challenging for erasure coding > on HDDs, but that's unrelated to rgw/you'd have the same problem with CephFS > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > >> >> Wido den Hollander <wido@xxxxxxxx> 于2020年5月12日周二 下午2:41写道： >> >> > >> > >> > On 5/12/20 4:22 AM, Zhenshi Zhou wrote: >> > > Hi all, >> > > >> > > We have several nfs servers providing file storage. There is a nginx >> in >> > > front of >> > > nfs servers in order to serve the clients. The files are mostly small >> > files >> > > and >> > > nearly about 30TB in total. >> > > >> > >> > What is small? How many objects/files are you talking about? >> > >> > > I'm gonna use ceph rgw as the storage. I wanna know if it's >> appropriate >> > to >> > > do so. >> > > The data migrating from nfs to rgw is a huge job. Besides I'm not sure >> > > whether >> > > ceph rgw is suitable in this scenario or not. >> > > >> > >> > Yes, it is. But make sure you don't put millions of objects into a >> > single bucket. Make sure that you spread them out so that you have let's >> > say 1M of objects per bucket at max. >> > >> > Wido >> > >> > > Thanks >> > > _______________________________________________ >> > > ceph-users mailing list -- ceph-users@xxxxxxx >> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > > >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |