Hi Boris,
Could you check something like
ceph daemon osd.23 perf dump | grep numpg
to see if there are some stray or removing PG?
Weiwen Hu
> 在 2021年4月15日，22:53，Boris Behrens <bb@xxxxxxxxx> 写道：
>
> Ah you are right.
> [root@s3db1 ~]# ceph daemon osd.23 config get bluestore_min_alloc_size_hdd
> {
> "bluestore_min_alloc_size_hdd": "65536"
> }
> But I also checked how many objects our s3 hold and the numbers just do not
> add up.
> There are only 26509200 objects, which would result in around 1TB "waste"
> if every object would be empty.
>
> I think the problem began when I updated the PG count from 1024 to 2048.
> Could there be an issue where the data is written twice?
>
>
>> Am Do., 15. Apr. 2021 um 16:48 Uhr schrieb Amit Ghadge <amitg.b14@xxxxxxxxx
>>> :
>>
>> verify those two parameter values ,bluestore_min_alloc_size_hdd &
>> bluestore_min_alloc_size_sdd, If you are using hdd disk then
>> bluestore_min_alloc_size_hdd are applicable.
>>
>>> On Thu, Apr 15, 2021 at 8:06 PM Boris Behrens <bb@xxxxxxxxx> wrote:
>>>
>>> So, I need to live with it? A value of zero leads to use the default?
>>> [root@s3db1 ~]# ceph daemon osd.23 config get bluestore_min_alloc_size
>>> {
>>> "bluestore_min_alloc_size": "0"
>>> }
>>>
>>> I also checked the fragmentation on the bluestore OSDs and it is around
>>> 0.80 - 0.89 on most OSDs. yikes.
>>> [root@s3db1 ~]# ceph daemon osd.23 bluestore allocator score block
>>> {
>>> "fragmentation_rating": 0.85906054329923576
>>> }
>>>
>>> The problem I currently have is, that I barely keep up with adding OSD
>>> disks.
>>>
>>> Am Do., 15. Apr. 2021 um 16:18 Uhr schrieb Amit Ghadge <
>>> amitg.b14@xxxxxxxxx>:
>>>
>>>> size_kb_actual are actually bucket object size but on OSD level the
>>>> bluestore_min_alloc_size default 64KB and SSD are 16KB
>>>>
>>>>
>>>> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faccess.redhat.com%2Fdocumentation%2Fen-us%2Fred_hat_ceph_storage%2F3%2Fhtml%2Fadministration_guide%2Fosd-bluestore&data=04%7C01%7C%7Cba98c0dff13941ea96ff08d9001e3759%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637540952043049058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wfSqqyiDHRXp4ypOGTxx4p%2Buy902OGPEmGkNfJ2BF6I%3D&reserved=0
>>>>
>>>> -AmitG
>>>>
>>>> On Thu, Apr 15, 2021 at 7:29 PM Boris Behrens <bb@xxxxxxxxx> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> maybe it is just a problem in my understanding, but it looks like our s3
>>>>> requires twice the space it should use.
>>>>>
>>>>> I ran "radosgw-admin bucket stats", and added all "size_kb_actual"
>>>>> values
>>>>> up and divided to TB (/1024/1024/1024).
>>>>> The resulting space is 135,1636733 TB. When I tripple it because of
>>>>> replication I end up with around 405TB which is nearly half the space of
>>>>> what ceph df tells me.
>>>>>
>>>>> Hope someone can help me.
>>>>>
>>>>> ceph df shows
>>>>> RAW STORAGE:
>>>>> CLASS SIZE AVAIL USED RAW USED %RAW
>>>>> USED
>>>>> hdd 1009 TiB 189 TiB 820 TiB 820 TiB
>>>>> 81.26
>>>>> TOTAL 1009 TiB 189 TiB 820 TiB 820 TiB
>>>>> 81.26
>>>>>
>>>>> POOLS:
>>>>> POOL ID PGS STORED
>>>>> OBJECTS
>>>>> USED %USED MAX AVAIL
>>>>> rbd 0 64 0 B
>>>>> 0
>>>>> 0 B 0 18 TiB
>>>>> .rgw.root 1 64 99 KiB
>>>>> 119
>>>>> 99 KiB 0 18 TiB
>>>>> eu-central-1.rgw.control 2 64 0 B
>>>>> 8
>>>>> 0 B 0 18 TiB
>>>>> eu-central-1.rgw.data.root 3 64 1.0 MiB
>>>>> 3.15k
>>>>> 1.0 MiB 0 18 TiB
>>>>> eu-central-1.rgw.gc 4 64 71 MiB
>>>>> 32
>>>>> 71 MiB 0 18 TiB
>>>>> eu-central-1.rgw.log 5 64 267 MiB
>>>>> 564
>>>>> 267 MiB 0 18 TiB
>>>>> eu-central-1.rgw.users.uid 6 64 2.8 MiB
>>>>> 6.91k
>>>>> 2.8 MiB 0 18 TiB
>>>>> eu-central-1.rgw.users.keys 7 64 263 KiB
>>>>> 6.73k
>>>>> 263 KiB 0 18 TiB
>>>>> eu-central-1.rgw.meta 8 64 384 KiB
>>>>> 1k
>>>>> 384 KiB 0 18 TiB
>>>>> eu-central-1.rgw.users.email 9 64 40 B
>>>>> 1
>>>>> 40 B 0 18 TiB
>>>>> eu-central-1.rgw.buckets.index 10 64 10 GiB
>>>>> 67.61k
>>>>> 10 GiB 0.02 18 TiB
>>>>> eu-central-1.rgw.buckets.data 11 2048 264 TiB
>>>>> 138.31M
>>>>> 264 TiB 83.37 18 TiB
>>>>> eu-central-1.rgw.buckets.non-ec 12 64 297 MiB
>>>>> 11.32k
>>>>> 297 MiB 0 18 TiB
>>>>> eu-central-1.rgw.usage 13 64 536 MiB
>>>>> 32
>>>>> 536 MiB 0 18 TiB
>>>>> eu-msg-1.rgw.control 56 64 0 B
>>>>> 8
>>>>> 0 B 0 18 TiB
>>>>> eu-msg-1.rgw.data.root 57 64 72 KiB
>>>>> 227
>>>>> 72 KiB 0 18 TiB
>>>>> eu-msg-1.rgw.gc 58 64 300 KiB
>>>>> 32
>>>>> 300 KiB 0 18 TiB
>>>>> eu-msg-1.rgw.log 59 64 835 KiB
>>>>> 242
>>>>> 835 KiB 0 18 TiB
>>>>> eu-msg-1.rgw.users.uid 60 64 56 KiB
>>>>> 104
>>>>> 56 KiB 0 18 TiB
>>>>> eu-msg-1.rgw.usage 61 64 37 MiB
>>>>> 25
>>>>> 37 MiB 0 18 TiB
>>>>> eu-msg-1.rgw.users.keys 62 64 3.8 KiB
>>>>> 97
>>>>> 3.8 KiB 0 18 TiB
>>>>> eu-msg-1.rgw.meta 63 64 607 KiB
>>>>> 1.60k
>>>>> 607 KiB 0 18 TiB
>>>>> eu-msg-1.rgw.buckets.index 64 64 71 MiB
>>>>> 119
>>>>> 71 MiB 0 18 TiB
>>>>> eu-msg-1.rgw.users.email 65 64 0 B
>>>>> 0
>>>>> 0 B 0 18 TiB
>>>>> eu-msg-1.rgw.buckets.data 66 64 2.9 TiB
>>>>> 1.16M
>>>>> 2.9 TiB 5.30 18 TiB
>>>>> eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB
>>>>> 354
>>>>> 2.2 MiB 0 18 TiB
>>>>> default.rgw.control 69 32 0 B
>>>>> 8
>>>>> 0 B 0 18 TiB
>>>>> default.rgw.data.root 70 32 0 B
>>>>> 0
>>>>> 0 B 0 18 TiB
>>>>> default.rgw.gc 71 32 0 B
>>>>> 0
>>>>> 0 B 0 18 TiB
>>>>> default.rgw.log 72 32 0 B
>>>>> 0
>>>>> 0 B 0 18 TiB
>>>>> default.rgw.users.uid 73 32 0 B
>>>>> 0
>>>>> 0 B 0 18 TiB
>>>>> fra-1.rgw.control 74 32 0 B
>>>>> 8
>>>>> 0 B 0 18 TiB
>>>>> fra-1.rgw.meta 75 32 0 B
>>>>> 0
>>>>> 0 B 0 18 TiB
>>>>> fra-1.rgw.log 76 32 50 B
>>>>> 28
>>>>> 50 B 0 18 TiB
>>>>>
>>>>>
>>>>> --
>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
>>>>> groüen Saal.
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>>>
>>>>
>>>
>>> --
>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
>>> groüen Saal.
>>>
>>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
