Re: CEPH Cluster Usage Discrepancy

[Sergey Malinin <hell@xxxxxxxxxxx>]

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024589.html

On 21.10.2018, at 16:12, Waterbly, Dan <dan.waterbly@xxxxxxxxxx> wrote:

Awesome! Thanks Serian!

Do you know where the 64KB comes from? Can that be tuned down for a cluster holding smaller objects?

Get Outlook for iOS

On Sat, Oct 20, 2018 at 10:49 PM -0700, "Serkan Çoban" <cobanserkan@xxxxxxxxx> wrote:

you have 24M objects, not 2.4M.
Each object will eat 64KB of storage, so 24M objects uses 1.5TB storage.
Add 3x replication to that, it is 4.5TB

On Sat, Oct 20, 2018 at 11:47 PM Waterbly, Dan  wrote:
>
> Hi Jakub,
>
> No, my setup seems to be the same as yours. Our system is mainly for archiving loads of data. This data has to be stored forever and allow reads, albeit seldom considering the number of objects we will store vs the number of objects that ever will be requested.
>
> It just really seems odd that the metadata surrounding the 25M objects is so high.
>
> We have 144 osds on 9 storage nodes. Perhaps it makes perfect sense but I’d like to know why we are seeing what we are and how it all adds up.
>
> Thanks!
> Dan
>
> Get Outlook for iOS
>
>
>
> On Sat, Oct 20, 2018 at 12:36 PM -0700, "Jakub Jaszewski"  wrote:
>
>> Hi Dan,
>>
>> Did you configure block.wal/block.db as separate devices/partition (osd_scenario: non-collocated or lvm for clusters installed using ceph-ansbile playbooks )?
>>
>> I run Ceph version 13.2.1 with non-collocated data.db and have the same situation - the sum of block.db partitions' size is displayed as RAW USED in ceph df.
>> Perhaps it is not the case for collocated block.db/wal.
>>
>> Jakub
>>
>> On Sat, Oct 20, 2018 at 8:34 PM Waterbly, Dan  wrote:
>>>
>>> I get that, but isn’t 4TiB to track 2.45M objects excessive? These numbers seem very high to me.
>>>
>>> Get Outlook for iOS
>>>
>>>
>>>
>>> On Sat, Oct 20, 2018 at 10:27 AM -0700, "Serkan Çoban"  wrote:
>>>
>>>> 4.65TiB includes size of wal and db partitions too.
>>>> On Sat, Oct 20, 2018 at 7:45 PM Waterbly, Dan  wrote:
>>>> >
>>>> > Hello,
>>>> >
>>>> >
>>>> >
>>>> > I have inserted 2.45M 1,000 byte objects into my cluster (radosgw, 3x replication).
>>>> >
>>>> >
>>>> >
>>>> > I am confused by the usage ceph df is reporting and am hoping someone can shed some light on this. Here is what I see when I run ceph df
>>>> >
>>>> >
>>>> >
>>>> > GLOBAL:
>>>> >
>>>> >     SIZE        AVAIL       RAW USED     %RAW USED
>>>> >
>>>> >     1.02PiB     1.02PiB      4.65TiB          0.44
>>>> >
>>>> > POOLS:
>>>> >
>>>> >     NAME                                           ID     USED        %USED     MAX AVAIL     OBJECTS
>>>> >
>>>> >     .rgw.root                                      1      3.30KiB         0        330TiB           17
>>>> >
>>>> >     .rgw.buckets.data      2      22.9GiB         0        330TiB     24550943
>>>> >
>>>> >     default.rgw.control                            3           0B         0        330TiB            8
>>>> >
>>>> >     default.rgw.meta                               4         373B         0        330TiB            3
>>>> >
>>>> >     default.rgw.log                                5           0B         0        330TiB            0
>>>> >
>>>> >     .rgw.control           6           0B         0        330TiB            8
>>>> >
>>>> >     .rgw.meta              7      2.18KiB         0        330TiB           12
>>>> >
>>>> >     .rgw.log               8           0B         0        330TiB          194
>>>> >
>>>> >     .rgw.buckets.index     9           0B         0        330TiB         2560
>>>> >
>>>> >
>>>> >
>>>> > Why does my bucket pool report usage of 22.9GiB but my cluster as a whole is reporting 4.65TiB? There is nothing else on this cluster as it was just installed and configured.
>>>> >
>>>> >
>>>> >
>>>> > Thank you for your help with this.
>>>> >
>>>> >
>>>> >
>>>> > -Dan
>>>> >
>>>> >
>>>> >
>>>> > Dan Waterbly | Senior Application Developer | 509.235.7500 x225 | dan.waterbly@xxxxxxxxxx
>>>> >
>>>> > WASHINGTON STATE ARCHIVES | DIGITAL ARCHIVES
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > ceph-users mailing list
>>>> > ceph-users@xxxxxxxxxxxxxx
>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com