Re: nfs migrate to rgw

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 5/18/20 1:51 PM, Zhenshi Zhou wrote:
> Hi Wido,
> 
> I did a research on the nfs files. I found that it contains much
> pictures about 
> 50KB, and much video files around 30MB. The amount of the files is more than
> 1 million. Maybe I can find a way to seperate the files in more buckets
> so that 
> there is no more than 1M objects in each bucket. But how about the small
> files 
> around 50KB. Does rgw serve well on small files?

I would recommend using different buckets. What I've done in such cases
is use the year+month for sharding.

For example: video-2020-05

RGW can serve objects which are 50kB in size, but there is overhead
involved. Storing a lot of such small objects comes at a price of overhead.

Wido

> 
> Wido den Hollander <wido@xxxxxxxx <mailto:wido@xxxxxxxx>> 于2020年5月12
> 日周二 下午2:41写道:
> 
> 
> 
>     On 5/12/20 4:22 AM, Zhenshi Zhou wrote:
>     > Hi all,
>     >
>     > We have several nfs servers providing file storage. There is a
>     nginx in
>     > front of
>     > nfs servers in order to serve the clients. The files are mostly
>     small files
>     > and
>     > nearly about 30TB in total.
>     >
> 
>     What is small? How many objects/files are you talking about?
> 
>     > I'm gonna use ceph rgw as the storage. I wanna know if it's
>     appropriate to
>     > do so.
>     > The data migrating from nfs to rgw is a huge job. Besides I'm not sure
>     > whether
>     > ceph rgw is suitable in this scenario or not.
>     >
> 
>     Yes, it is. But make sure you don't put millions of objects into a
>     single bucket. Make sure that you spread them out so that you have let's
>     say 1M of objects per bucket at max.
> 
>     Wido
> 
>     > Thanks
>     > _______________________________________________
>     > ceph-users mailing list -- ceph-users@xxxxxxx
>     <mailto:ceph-users@xxxxxxx>
>     > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>     <mailto:ceph-users-leave@xxxxxxx>
>     >
> 
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux