Re: Size and capacity calculations questions

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Home directories probably means lots of small objects. Default minimum allocation size of BlueStore on HDD is 64 kiB, so there's a lot of overhead for everything smaller;

Details: google bluestore min alloc size, can only be changed during OSD creation

Paul

--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


On Fri, Dec 6, 2019 at 12:57 PM Jochen Schulz <schulz@xxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi!

Thank you!
The output of both commands are below.
I still dont understand why there are 21T used data (because 5.5T*3 =
16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the
osd output tells we have 25T free space.


$ sudo ceph df
RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED
    hdd        45 TiB      24 TiB      21 TiB       21 TiB         46.33
    ssd       596 GiB     524 GiB     1.7 GiB       72 GiB         12.09
    TOTAL      46 TiB      25 TiB      21 TiB       21 TiB         45.89

POOLS:
    POOL                ID     STORED      OBJECTS     USED        %USED
    MAX AVAIL
    images               8     149 GiB      38.30k     354 GiB      2.52
      4.5 TiB
    cephfs_data          9     5.5 TiB      26.61M      20 TiB     60.36
      4.5 TiB
    cephfs_metadata     10      12 GiB       3.17M      13 GiB      2.57
      164 GiB


$ sudo ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE DATA    OMAP    META    AVAIL
   %USE  VAR  PGS STATUS
 0   hdd 0.89000  1.00000 931 GiB 456 GiB 453 GiB 136 MiB 3.5 GiB  475
GiB 49.01 1.07 103     up
 1   hdd 0.89000  1.00000 931 GiB 495 GiB 491 GiB 100 MiB 3.9 GiB  436
GiB 53.14 1.16  89     up
 4   hdd 0.89000  1.00000 931 GiB 345 GiB 342 GiB 108 MiB 2.9 GiB  586
GiB 37.05 0.81  87     up
 5   hdd 0.89000  1.00000 931 GiB 521 GiB 517 GiB 108 MiB 4.1 GiB  410
GiB 55.96 1.22  98     up
 6   hdd 0.89000  1.00000 931 GiB 367 GiB 364 GiB  95 MiB 3.2 GiB  564
GiB 39.44 0.86  95     up
 7   hdd 0.89000  1.00000 931 GiB 540 GiB 536 GiB  77 MiB 3.7 GiB  392
GiB 57.96 1.26 111     up
20   hdd 0.89000  1.00000 931 GiB 382 GiB 378 GiB  60 MiB 3.3 GiB  550
GiB 40.96 0.89  85     up
23   hdd 1.81929  1.00000 1.8 TiB 706 GiB 701 GiB 113 MiB 4.9 GiB  1.1
TiB 37.92 0.83 182     up
44   hdd 0.89000  1.00000 931 GiB 468 GiB 465 GiB  34 MiB 3.3 GiB  463
GiB 50.29 1.10  93     up
45   hdd 1.78999  1.00000 1.8 TiB 882 GiB 875 GiB 138 MiB 6.3 GiB  981
GiB 47.33 1.03 179     up
46   hdd 1.78999  1.00000 1.8 TiB 910 GiB 903 GiB 127 MiB 6.4 GiB  953
GiB 48.83 1.06 192     up
22   ssd 0.11639  1.00000 119 GiB  15 GiB 357 MiB  12 GiB 2.8 GiB  104
GiB 12.61 0.27 315     up
12   hdd 0.89000  1.00000 931 GiB 499 GiB 494 GiB  64 MiB 4.5 GiB  432
GiB 53.57 1.17 116     up
13   hdd 0.89000  1.00000 931 GiB 536 GiB 532 GiB  48 MiB 4.4 GiB  395
GiB 57.59 1.26 109     up
30   hdd 0.89000  1.00000 931 GiB 510 GiB 506 GiB  33 MiB 3.9 GiB  421
GiB 54.80 1.19 100     up
32   hdd 0.89000  1.00000 931 GiB 495 GiB 491 GiB  56 MiB 4.1 GiB  436
GiB 53.17 1.16 101     up
33   hdd 0.89000  1.00000 931 GiB 333 GiB 330 GiB  56 MiB 3.1 GiB  598
GiB 35.80 0.78  82     up
15   ssd 0.11639  1.00000 119 GiB  14 GiB 336 MiB  11 GiB 2.9 GiB  105
GiB 12.13 0.26 305     up
17   hdd 0.89000  1.00000 931 GiB 577 GiB 573 GiB  77 MiB 4.4 GiB  354
GiB 61.99 1.35  97     up
18   hdd 0.89000  1.00000 931 GiB 413 GiB 409 GiB  70 MiB 4.0 GiB  518
GiB 44.34 0.97  95     up
19   hdd 1.81879  1.00000 1.8 TiB 895 GiB 889 GiB 144 MiB 5.6 GiB  967
GiB 48.06 1.05 184     up
21   hdd 0.89000  1.00000 931 GiB 360 GiB 357 GiB  60 MiB 3.4 GiB  570
GiB 38.72 0.84 100     up
31   hdd 0.90909  1.00000 931 GiB 508 GiB 505 GiB  80 MiB 3.5 GiB  423
GiB 54.58 1.19 102     up
25   ssd 0.11639  1.00000 119 GiB  14 GiB 339 MiB  11 GiB 2.7 GiB  105
GiB 11.86 0.26 310     up
 8   hdd 0.89000  1.00000 931 GiB 359 GiB 356 GiB  72 MiB 3.1 GiB  572
GiB 38.55 0.84  80     up
 9   hdd 0.89000  1.00000 931 GiB 376 GiB 373 GiB  42 MiB 3.0 GiB  555
GiB 40.39 0.88  87     up
24   hdd 0.89000  1.00000 931 GiB 342 GiB 339 GiB  70 MiB 2.8 GiB  590
GiB 36.69 0.80  78     up
26   hdd 1.78999  1.00000 1.8 TiB 921 GiB 915 GiB 129 MiB 6.1 GiB  942
GiB 49.45 1.08 177     up
27   hdd 1.78999  1.00000 1.8 TiB 891 GiB 885 GiB 125 MiB 5.7 GiB  972
GiB 47.82 1.04 208     up
35   hdd 1.81929  1.00000 1.8 TiB 819 GiB 814 GiB 110 MiB 5.3 GiB  1.0
TiB 43.99 0.96 184     up
29   ssd 0.11638  1.00000 119 GiB  15 GiB 339 MiB  11 GiB 2.9 GiB  105
GiB 12.25 0.27 311     up
14   hdd 1.81929  1.00000 1.8 TiB 889 GiB 884 GiB 1.9 MiB 4.3 GiB  974
GiB 47.70 1.04 162     up
28   hdd 1.81929  1.00000 1.8 TiB 609 GiB 606 GiB 3.4 MiB 2.9 GiB  1.2
TiB 32.67 0.71 169     up
36   hdd 1.36429  1.00000 1.4 TiB 591 GiB 589 GiB 2.0 MiB 2.9 GiB  806
GiB 42.34 0.92 135     up
37   hdd 1.81929  1.00000 1.8 TiB 840 GiB 836 GiB 2.5 MiB 4.1 GiB 1023
GiB 45.12 0.98 177     up
38   hdd 1.81929  1.00000 1.8 TiB 914 GiB 909 GiB 2.2 MiB 4.4 GiB  949
GiB 49.05 1.07 182     up
 2   hdd 0.89000  1.00000 931 GiB 449 GiB 445 GiB  60 MiB 3.4 GiB  482
GiB 48.19 1.05  89     up
 3   hdd 0.89000  1.00000 931 GiB 333 GiB 329 GiB  75 MiB 3.0 GiB  598
GiB 35.73 0.78  86     up
10   hdd 0.89000  1.00000 931 GiB 416 GiB 412 GiB  54 MiB 3.8 GiB  515
GiB 44.68 0.97 100     up
11   hdd 1.81879  1.00000 1.8 TiB 803 GiB 798 GiB  70 MiB 5.3 GiB  1.0
TiB 43.14 0.94 188     up
34   hdd 0.90909  1.00000 931 GiB 605 GiB 600 GiB  64 MiB 4.2 GiB  326
GiB 64.95 1.42 106     up
16   ssd 0.11639  1.00000 119 GiB  14 GiB 322 MiB  10 GiB 3.1 GiB  105
GiB 11.59 0.25 295     up
                    TOTAL  46 TiB  21 TiB  21 TiB  59 GiB 165 GiB   25
TiB 45.89
MIN/MAX VAR: 0.25/1.42  STDDEV: 13.76




On 06.12.19 12:27, Aleksey Gutikov wrote:
> On 6.12.19 13:29, Jochen Schulz wrote:
>> Hi!
>>
>> We have a ceph cluster with 42 OSD in production as a server providing
>> mainly home-directories of users. Ceph is 14.2.4 nautilus.
>>
>> We have 3 pools. One images (for rbd images) a cephfs_metadata and a
>> cephfs_data pool.
>>
>> Our raw data is about 5.6T. All pools have replica size 3 and there are
>> only very little snapshots in the rbd images pool, the cephfspool doesnt
>> use snapshots.
>>
>> How is it possible that the status tells us, that 21T/46T  is used,
>> because thats much more than 3 times the raw size.
>>
>> Also, to make that more confusing, there as at least half of the cluster
>> free, and we get pg backfill_toofull after we added some OSDs lately.
>> The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T
>> free.
>>
>> The autoscale module seems to calculate the 20T times 3 for the space
>> needed and thus has wrong numbers (see below).
>>
>> Status of the cluster is added below too.
>>
>> how can these size/capacity numbers be explained?
>> and, would be there a recommendation to change something?
>>
>> Thank you in advance!
>>
>> best
>> Jochen
>>
>>
>> # ceph -s
>>
>>   cluster:
>>      id:     2b16167f-3f33-4580-a0e9-7a71978f403d
>>      health: HEALTH_ERR
>>              Degraded data redundancy (low space): 1 pg backfill_toofull
>>              1 subtrees have overcommitted pool target_size_bytes
>>              1 subtrees have overcommitted pool target_size_ratio
>>              2 pools have too many placement groups
>>
>>    services:
>>      mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d)
>>      mgr: earl(active, since 24h), standbys: assam
>>      mds: cephfs:1 {0=assam=up:active} 1 up:standby
>>      osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped
>> pgs
>>
>>    data:
>>      pools:   3 pools, 2048 pgs
>>      objects: 29.80M objects, 5.6 TiB
>>      usage:   21 TiB used, 25 TiB / 46 TiB avail
>>      pgs:     1164396/89411013 objects misplaced (1.302%)
>>               2018 active+clean
>>               22   active+remapped+backfill_wait
>>               7    active+remapped+backfilling
>>               1    active+remapped+backfill_wait+backfill_toofull
>>
>>    io:
>>      client:   1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr
>>      recovery: 9.2 MiB/s, 41 objects/s
>>
>>
>> # ceph osd pool autoscale-status
>>   POOL               SIZE  TARGET SIZE  RATE  RAW CAPACITY   RATIO
>> TARGET RATIO  BIAS  PG_NUM  NEW PG_NUM  AUTOSCALE
>>   images           354.2G                3.0        46100G  0.0231
>>           1.0    1024          32  warn
>>   cephfs_metadata  13260M                3.0        595.7G  0.0652
>>           1.0     512           8  warn
>>   cephfs_data      20802G                3.0        46100G  1.3537
>>           1.0     512              warn
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>
> Please, provide output of ceph df and ceph osd df - that should explain
> both questions (21T and 82%).
>
>

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux