Re: Using RBD to pack billions of small files

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

On 04/02/2021 07:41, Loïc Dachary wrote:

On 04/02/2021 05:51, Federico Lucifredi wrote:
Hi Loïc,
    I am intrigued, but am missing something: why not using RGW, and store the source code files as objects? RGW has native compression and can take care of that behind the scenes.
Excellent question!

    Is the desire to use RBD only due to minimum allocation sizes?
I *assume* that since RGW does have specific strategies to take advantage of the fact that objects are immutable and will never be removed:

* It will be slower to add artifacts in RGW than in an RBD image + index
* The metadata in RGW will be larger than an RBD image + index

RGW addition is pretty quick up to fairly large buckets; and if you're not expecting to want to list the bucket contents often, then RGW might well be a good option for your object store with small files.

Or at least, using some of the RGW code (I think there's a librgw) to re-use a bunch of its code for your use case; this feels more natural to me than using RBD for this.

Regards,

Matthew
[pleased software heritage are still looking at Ceph :) ]


--
The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux