On Mon, May 18, 2020 at 1:52 PM Zhenshi Zhou <deaderzzs@xxxxxxxxx> wrote: > > 50KB, and much video files around 30MB. The amount of the files is more > than > 1 million. Maybe I can find a way to seperate the files in more buckets so > that > there is no more than 1M objects in each bucket. But how about the small > files > around 50KB. Does rgw serve well on small files? > 1 million files is usually the point where you first need to start thinking about some optimizations, but that's mostly just making sure that the index is on SSD and it'll happily work up to ~10 million files. Then you might need to start thinking about the index being on *good* SSDs (and/or on many SSDs/DB devices). It starts the get interesting if you need to go beyond 100 million files, that's the point where you need to start tuning shard sizes and the types of index queries that you send... I've found that a few hundred million objects per bucket are no problem if you run with large shard sizes (500k - 1 million); however, there are some index-queries that can be really expensive like filtering on prefixes in some pathological cases... Small files: sure, works well, but can be challenging for erasure coding on HDDs, but that's unrelated to rgw/you'd have the same problem with CephFS Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 > > Wido den Hollander <wido@xxxxxxxx> 于2020年5月12日周二 下午2:41写道： > > > > > > > On 5/12/20 4:22 AM, Zhenshi Zhou wrote: > > > Hi all, > > > > > > We have several nfs servers providing file storage. There is a nginx in > > > front of > > > nfs servers in order to serve the clients. The files are mostly small > > files > > > and > > > nearly about 30TB in total. > > > > > > > What is small? How many objects/files are you talking about? > > > > > I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate > > to > > > do so. > > > The data migrating from nfs to rgw is a huge job. Besides I'm not sure > > > whether > > > ceph rgw is suitable in this scenario or not. > > > > > > > Yes, it is. But make sure you don't put millions of objects into a > > single bucket. Make sure that you spread them out so that you have let's > > say 1M of objects per bucket at max. > > > > Wido > > > > > Thanks > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |