Home directories probably means lots of small objects. Default minimum allocation size of BlueStore on HDD is 64 kiB, so there's a lot of overhead for everything smaller;
Details: google bluestore min alloc size, can only be changed during OSD creation
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Dec 6, 2019 at 12:57 PM Jochen Schulz <schulz@xxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi!
Thank you!
The output of both commands are below.
I still dont understand why there are 21T used data (because 5.5T*3 =
16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the
osd output tells we have 25T free space.
$ sudo ceph df
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 45 TiB 24 TiB 21 TiB 21 TiB 46.33
ssd 596 GiB 524 GiB 1.7 GiB 72 GiB 12.09
TOTAL 46 TiB 25 TiB 21 TiB 21 TiB 45.89
POOLS:
POOL ID STORED OBJECTS USED %USED
MAX AVAIL
images 8 149 GiB 38.30k 354 GiB 2.52
4.5 TiB
cephfs_data 9 5.5 TiB 26.61M 20 TiB 60.36
4.5 TiB
cephfs_metadata 10 12 GiB 3.17M 13 GiB 2.57
164 GiB
$ sudo ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS
0 hdd 0.89000 1.00000 931 GiB 456 GiB 453 GiB 136 MiB 3.5 GiB 475
GiB 49.01 1.07 103 up
1 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 100 MiB 3.9 GiB 436
GiB 53.14 1.16 89 up
4 hdd 0.89000 1.00000 931 GiB 345 GiB 342 GiB 108 MiB 2.9 GiB 586
GiB 37.05 0.81 87 up
5 hdd 0.89000 1.00000 931 GiB 521 GiB 517 GiB 108 MiB 4.1 GiB 410
GiB 55.96 1.22 98 up
6 hdd 0.89000 1.00000 931 GiB 367 GiB 364 GiB 95 MiB 3.2 GiB 564
GiB 39.44 0.86 95 up
7 hdd 0.89000 1.00000 931 GiB 540 GiB 536 GiB 77 MiB 3.7 GiB 392
GiB 57.96 1.26 111 up
20 hdd 0.89000 1.00000 931 GiB 382 GiB 378 GiB 60 MiB 3.3 GiB 550
GiB 40.96 0.89 85 up
23 hdd 1.81929 1.00000 1.8 TiB 706 GiB 701 GiB 113 MiB 4.9 GiB 1.1
TiB 37.92 0.83 182 up
44 hdd 0.89000 1.00000 931 GiB 468 GiB 465 GiB 34 MiB 3.3 GiB 463
GiB 50.29 1.10 93 up
45 hdd 1.78999 1.00000 1.8 TiB 882 GiB 875 GiB 138 MiB 6.3 GiB 981
GiB 47.33 1.03 179 up
46 hdd 1.78999 1.00000 1.8 TiB 910 GiB 903 GiB 127 MiB 6.4 GiB 953
GiB 48.83 1.06 192 up
22 ssd 0.11639 1.00000 119 GiB 15 GiB 357 MiB 12 GiB 2.8 GiB 104
GiB 12.61 0.27 315 up
12 hdd 0.89000 1.00000 931 GiB 499 GiB 494 GiB 64 MiB 4.5 GiB 432
GiB 53.57 1.17 116 up
13 hdd 0.89000 1.00000 931 GiB 536 GiB 532 GiB 48 MiB 4.4 GiB 395
GiB 57.59 1.26 109 up
30 hdd 0.89000 1.00000 931 GiB 510 GiB 506 GiB 33 MiB 3.9 GiB 421
GiB 54.80 1.19 100 up
32 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 56 MiB 4.1 GiB 436
GiB 53.17 1.16 101 up
33 hdd 0.89000 1.00000 931 GiB 333 GiB 330 GiB 56 MiB 3.1 GiB 598
GiB 35.80 0.78 82 up
15 ssd 0.11639 1.00000 119 GiB 14 GiB 336 MiB 11 GiB 2.9 GiB 105
GiB 12.13 0.26 305 up
17 hdd 0.89000 1.00000 931 GiB 577 GiB 573 GiB 77 MiB 4.4 GiB 354
GiB 61.99 1.35 97 up
18 hdd 0.89000 1.00000 931 GiB 413 GiB 409 GiB 70 MiB 4.0 GiB 518
GiB 44.34 0.97 95 up
19 hdd 1.81879 1.00000 1.8 TiB 895 GiB 889 GiB 144 MiB 5.6 GiB 967
GiB 48.06 1.05 184 up
21 hdd 0.89000 1.00000 931 GiB 360 GiB 357 GiB 60 MiB 3.4 GiB 570
GiB 38.72 0.84 100 up
31 hdd 0.90909 1.00000 931 GiB 508 GiB 505 GiB 80 MiB 3.5 GiB 423
GiB 54.58 1.19 102 up
25 ssd 0.11639 1.00000 119 GiB 14 GiB 339 MiB 11 GiB 2.7 GiB 105
GiB 11.86 0.26 310 up
8 hdd 0.89000 1.00000 931 GiB 359 GiB 356 GiB 72 MiB 3.1 GiB 572
GiB 38.55 0.84 80 up
9 hdd 0.89000 1.00000 931 GiB 376 GiB 373 GiB 42 MiB 3.0 GiB 555
GiB 40.39 0.88 87 up
24 hdd 0.89000 1.00000 931 GiB 342 GiB 339 GiB 70 MiB 2.8 GiB 590
GiB 36.69 0.80 78 up
26 hdd 1.78999 1.00000 1.8 TiB 921 GiB 915 GiB 129 MiB 6.1 GiB 942
GiB 49.45 1.08 177 up
27 hdd 1.78999 1.00000 1.8 TiB 891 GiB 885 GiB 125 MiB 5.7 GiB 972
GiB 47.82 1.04 208 up
35 hdd 1.81929 1.00000 1.8 TiB 819 GiB 814 GiB 110 MiB 5.3 GiB 1.0
TiB 43.99 0.96 184 up
29 ssd 0.11638 1.00000 119 GiB 15 GiB 339 MiB 11 GiB 2.9 GiB 105
GiB 12.25 0.27 311 up
14 hdd 1.81929 1.00000 1.8 TiB 889 GiB 884 GiB 1.9 MiB 4.3 GiB 974
GiB 47.70 1.04 162 up
28 hdd 1.81929 1.00000 1.8 TiB 609 GiB 606 GiB 3.4 MiB 2.9 GiB 1.2
TiB 32.67 0.71 169 up
36 hdd 1.36429 1.00000 1.4 TiB 591 GiB 589 GiB 2.0 MiB 2.9 GiB 806
GiB 42.34 0.92 135 up
37 hdd 1.81929 1.00000 1.8 TiB 840 GiB 836 GiB 2.5 MiB 4.1 GiB 1023
GiB 45.12 0.98 177 up
38 hdd 1.81929 1.00000 1.8 TiB 914 GiB 909 GiB 2.2 MiB 4.4 GiB 949
GiB 49.05 1.07 182 up
2 hdd 0.89000 1.00000 931 GiB 449 GiB 445 GiB 60 MiB 3.4 GiB 482
GiB 48.19 1.05 89 up
3 hdd 0.89000 1.00000 931 GiB 333 GiB 329 GiB 75 MiB 3.0 GiB 598
GiB 35.73 0.78 86 up
10 hdd 0.89000 1.00000 931 GiB 416 GiB 412 GiB 54 MiB 3.8 GiB 515
GiB 44.68 0.97 100 up
11 hdd 1.81879 1.00000 1.8 TiB 803 GiB 798 GiB 70 MiB 5.3 GiB 1.0
TiB 43.14 0.94 188 up
34 hdd 0.90909 1.00000 931 GiB 605 GiB 600 GiB 64 MiB 4.2 GiB 326
GiB 64.95 1.42 106 up
16 ssd 0.11639 1.00000 119 GiB 14 GiB 322 MiB 10 GiB 3.1 GiB 105
GiB 11.59 0.25 295 up
TOTAL 46 TiB 21 TiB 21 TiB 59 GiB 165 GiB 25
TiB 45.89
MIN/MAX VAR: 0.25/1.42 STDDEV: 13.76
On 06.12.19 12:27, Aleksey Gutikov wrote:
> On 6.12.19 13:29, Jochen Schulz wrote:
>> Hi!
>>
>> We have a ceph cluster with 42 OSD in production as a server providing
>> mainly home-directories of users. Ceph is 14.2.4 nautilus.
>>
>> We have 3 pools. One images (for rbd images) a cephfs_metadata and a
>> cephfs_data pool.
>>
>> Our raw data is about 5.6T. All pools have replica size 3 and there are
>> only very little snapshots in the rbd images pool, the cephfspool doesnt
>> use snapshots.
>>
>> How is it possible that the status tells us, that 21T/46T is used,
>> because thats much more than 3 times the raw size.
>>
>> Also, to make that more confusing, there as at least half of the cluster
>> free, and we get pg backfill_toofull after we added some OSDs lately.
>> The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T
>> free.
>>
>> The autoscale module seems to calculate the 20T times 3 for the space
>> needed and thus has wrong numbers (see below).
>>
>> Status of the cluster is added below too.
>>
>> how can these size/capacity numbers be explained?
>> and, would be there a recommendation to change something?
>>
>> Thank you in advance!
>>
>> best
>> Jochen
>>
>>
>> # ceph -s
>>
>> cluster:
>> id: 2b16167f-3f33-4580-a0e9-7a71978f403d
>> health: HEALTH_ERR
>> Degraded data redundancy (low space): 1 pg backfill_toofull
>> 1 subtrees have overcommitted pool target_size_bytes
>> 1 subtrees have overcommitted pool target_size_ratio
>> 2 pools have too many placement groups
>>
>> services:
>> mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d)
>> mgr: earl(active, since 24h), standbys: assam
>> mds: cephfs:1 {0=assam=up:active} 1 up:standby
>> osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped
>> pgs
>>
>> data:
>> pools: 3 pools, 2048 pgs
>> objects: 29.80M objects, 5.6 TiB
>> usage: 21 TiB used, 25 TiB / 46 TiB avail
>> pgs: 1164396/89411013 objects misplaced (1.302%)
>> 2018 active+clean
>> 22 active+remapped+backfill_wait
>> 7 active+remapped+backfilling
>> 1 active+remapped+backfill_wait+backfill_toofull
>>
>> io:
>> client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr
>> recovery: 9.2 MiB/s, 41 objects/s
>>
>>
>> # ceph osd pool autoscale-status
>> POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO
>> TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE
>> images 354.2G 3.0 46100G 0.0231
>> 1.0 1024 32 warn
>> cephfs_metadata 13260M 3.0 595.7G 0.0652
>> 1.0 512 8 warn
>> cephfs_data 20802G 3.0 46100G 1.3537
>> 1.0 512 warn
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>
> Please, provide output of ceph df and ceph osd df - that should explain
> both questions (21T and 82%).
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
