ceph-df free discrepancy

[Reed Dier <reed.dier@xxxxxxxxxxx>]

Hopefully someone can sanity check me here, but I'm getting the feeling that the MAX AVAIL in ceph df isn't reporting the correct value in 14.2.8 (mon/mgr/mds are .8, most OSDs are .7)

RAW STORAGE:

    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED

    hdd       530 TiB     163 TiB     366 TiB      367 TiB         69.19

    ssd       107 TiB      37 TiB      70 TiB       70 TiB         65.33

    TOTAL     637 TiB     201 TiB     436 TiB      437 TiB         68.54

POOLS:

    POOL                      ID     STORED      OBJECTS     USED        %USED     MAX AVAIL

    fs-metadata               16      44 GiB       4.16M      44 GiB      0.25       5.6 TiB

    cephfs-hdd-3x             17      46 TiB     109.54M     144 TiB     61.81        30 TiB

    objects-hybrid            20      46 TiB     537.08M     187 TiB     91.71       5.6 TiB

    objects-hdd               24     224 GiB      50.81k     676 GiB      0.74        30 TiB

    rbd-hybrid                29     3.8 TiB       1.19M      11 TiB     40.38       5.6 TiB

    device_health_metrics     33     270 MiB         327     270 MiB         0        30 TiB

    rbd-ssd                   34     4.2 TiB       1.19M      12 TiB     41.55       5.6 TiB

    cephfs-hdd-ec73           37      42 TiB      30.35M      72 TiB     44.78        62 TiB

I have a few pools that don't look like they are calculating the available storage for that pool correctly.

Specifically, any of my hybrid pools (20,29) or all-SSD pools (16,34).

For my hybrid pools, I have a crush rule of take 1 of host in the ssd root, take -1 chassis in the hdd root.

For my ssd pools, I have a crush rule of take 0 of host in the ssd root.

Now I have 60 ssd osds 1.92T each, and sadly distribution is imperfect (leaving those issues out of this), and I have plenty of underfull and overfull osds, which I am trying to manually reweighs to get my most full's down to free up space:

$ ceph osd df class ssd | sort -k17

ID  CLASS WEIGHT  REWEIGHT SIZE    RAW USE DATA     OMAP    META    AVAIL   %USE  VAR  PGS STATUS

MIN/MAX VAR: 0.80/1.21  STDDEV: 5.56

                     TOTAL 107 TiB  70 TiB   68 TiB 163 GiB 431 GiB  37 TiB 65.33

 28   ssd 1.77879  1.00000 1.8 TiB 951 GiB  916 GiB 7.1 GiB 5.8 GiB 871 GiB 52.20 0.80  68     up

 33   ssd 1.77879  1.00000 1.8 TiB 1.0 TiB 1010 GiB 6.0 MiB 5.9 GiB 777 GiB 57.33 0.88  74     up

 47   ssd 1.77879  1.00000 1.8 TiB 1.0 TiB 1011 GiB 6.7 GiB 6.4 GiB 776 GiB 57.38 0.88  75     up

[SNIP]

 57   ssd 1.77879  0.98000 1.8 TiB 1.4 TiB  1.3 TiB 6.2 GiB 8.6 GiB 417 GiB 77.08 1.18 102     up

107   ssd 1.80429  1.00000 1.8 TiB 1.4 TiB  1.3 TiB 7.0 GiB 8.7 GiB 422 GiB 77.15 1.18 102     up

 50   ssd 1.77879  1.00000 1.8 TiB 1.4 TiB  1.4 TiB 5.5 MiB 8.6 GiB 381 GiB 79.10 1.21 105     up

 60   ssd 1.77879  0.92000 1.8 TiB 1.4 TiB  1.4 TiB 6.2 MiB 9.0 GiB 379 GiB 79.17 1.21 105     up

That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free, if all OSD's grew evenly, which they won't, which is still far short of 37TiB raw free, as expected.

However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would be 22.8/3=7.6TiB.

But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated data consumes.

Meaning if my napkin math is right, should show ~22.8TiB free.

Am I grossly mis-understanding how this is calculated?

Maybe this is fixed in Octopus?

Just trying to get a grasp on what I'm seeing not matching expectations.

Thanks,

Reed

Attachment: smime.p7s
Description: S/MIME cryptographic signature

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx