On Thu, Feb 25, 2021 at 10:55:05AM +0000, Simon Sutter wrote: > The output of ceph df detail is: > --- RAW STORAGE --- > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 > TOTAL 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 > > --- POOLS --- > POOL ID PGS STORED OBJECTS %USED MAX AVAIL > device_health_metrics 1 1 51 MiB 48 0 30 TiB > rep_data_fs 2 32 14 KiB 3.41k 0 30 TiB > rep_meta_fs 3 32 227 MiB 1.72k 0 30 TiB > ec_bkp1 4 32 4.2 TiB 1.10M 6.11 67 TiB > > So ec_bkp1 uses 4.2TiB an there are 67TiB free usable Storage. > This means total EC usable storage would be 71.2TiB. > But calculating with the 109TiB RAW storage, shouldn't it be 81.75? The "MAX AVAIL" is not "free", i.e. it is not a difference between total and used. It is estimated by the cluster as "how much data user will be able to store additionally until one of osds is filled up". Consider an example, when by some reason all osds are 10% full and one 50% full. The cluster will "assume" that if you store additional data it will be distributed the same way, i.e. that one osd will continue to store more than others, so when all other osds will 20% full that one will be 100% full and the cluster is not usable. I.e. basically "MAX AVAIL" in this example is (N - 1) * 10% + 1 * 50%, instead of (N - 1) * 90% + 1 * 50%, which whould you expect for "free". To make "MAX AVAIL" match exactly "free", you have to have a perfectly balanced cluster. Look at `ceph osd df` output to see how well data is balanced in your case. -- Mykola Golub _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |