Hi All,
Does anyone know how I can rebalance my cluster to balance out the OSD
usage?
I just added 12 more 14Tb HDDs to my cluster (cluster of made up of 12Tb
and 14Tb disks) bringing my total to 48 OSDs. Ceph df reports my pool as 83%
full (see below). I am aware this only reports the fullest drive. However,
after backfilling, data distribution is still heavily skewed to a few OSDs.
The oldest are at ~80% and the newest are at ~50. My issue is that data is
currently being written to all drives equally and at this rate the cluster
will start disabling writes despite still having significant raw capacity.
PGs do not seem to be evenly distributed either. Weights are 12.73 for all
14Tb OSD and 10.9 for all 12Tb OSDs. I tried using the balancer to create a
new plan but it had no real effect. Haven't seen much useful info online.
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 582 TiB 203 TiB 379 TiB 380 TiB 65.20
TOTAL 582 TiB 203 TiB 379 TiB 380 TiB 65.20
--- POOLS ---
POOL ID STORED OBJECTS USED %USED MAX AVAIL
cephfs_data 2 261 TiB 68.62M 379 TiB 83.64 56 TiB
cephfs_metadata 3 536 MiB 12.74k 1.6 GiB 0 25 TiB
`osd df`:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS
0 hdd 10.91409 1.00000 11 TiB 8.4 TiB 8.4 TiB 1.6 MiB 22 GiB
2.5 TiB 77.00 1.18 92 up
4 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 21 MiB 18 GiB
4.0 TiB 63.03 0.97 80 up
5 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.3 MiB 17 GiB
4.4 TiB 59.50 0.91 71 up
11 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.4 TiB 36 MiB 24 GiB
3.3 TiB 74.13 1.14 105 up
15 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.2 TiB 20 MiB 21 GiB
4.5 TiB 64.75 0.99 99 up
16 hdd 10.91409 1.00000 11 TiB 6.4 TiB 6.4 TiB 1.7 MiB 16 GiB
4.5 TiB 58.81 0.90 74 up
20 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 33 MiB 23 GiB
4.3 TiB 66.09 1.01 102 up
22 hdd 12.73340 1.00000 13 TiB 8.6 TiB 8.6 TiB 3.3 MiB 22 GiB
4.1 TiB 67.48 1.04 100 up
23 hdd 12.73340 1.00000 13 TiB 7.6 TiB 7.6 TiB 406 KiB 20 GiB
5.1 TiB 59.93 0.92 89 up
36 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.2 TiB 171 KiB 16 GiB
5.5 TiB 57.00 0.87 85 up
41 hdd 12.73340 1.00000 13 TiB 7.0 TiB 7.0 TiB 2.1 MiB 16 GiB
5.7 TiB 54.85 0.84 85 up
46 hdd 12.73340 1.00000 13 TiB 8.1 TiB 8.0 TiB 67 MiB 19 GiB
4.7 TiB 63.34 0.97 99 up
3 hdd 10.91409 1.00000 11 TiB 7.6 TiB 7.6 TiB 1.0 MiB 19 GiB
3.3 TiB 69.85 1.07 87 up
6 hdd 10.91409 1.00000 11 TiB 8.3 TiB 8.3 TiB 39 MiB 22 GiB
2.6 TiB 76.28 1.17 93 up
10 hdd 12.73340 1.00000 13 TiB 9.6 TiB 9.6 TiB 5.1 MiB 24 GiB
3.1 TiB 75.52 1.16 109 up
14 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.8 TiB 34 MiB 22 GiB
3.9 TiB 69.57 1.07 99 up
24 hdd 12.73340 1.00000 13 TiB 11 TiB 11 TiB 22 MiB 34 GiB
2.1 TiB 83.28 1.28 117 up
25 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 24 MiB 21 GiB
4.4 TiB 65.84 1.01 97 up
26 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.9 TiB 205 KiB 22 GiB
3.8 TiB 69.84 1.07 102 up
30 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 33 MiB 16 GiB
4.0 TiB 63.31 0.97 92 up
39 hdd 12.73340 1.00000 13 TiB 7.2 TiB 7.2 TiB 36 MiB 16 GiB
5.6 TiB 56.39 0.86 93 up
43 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 3.6 MiB 19 GiB
4.4 TiB 65.54 1.01 98 up
47 hdd 12.73340 1.00000 13 TiB 7.1 TiB 7.1 TiB 51 MiB 17 GiB
5.6 TiB 55.70 0.85 88 up
2 hdd 10.91409 1.00000 11 TiB 7.4 TiB 7.3 TiB 22 KiB 19 GiB
3.6 TiB 67.36 1.03 82 up
7 hdd 10.91409 1.00000 11 TiB 8.5 TiB 8.5 TiB 530 KiB 21 GiB
2.4 TiB 77.89 1.19 97 up
8 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 21 MiB 22 GiB
4.3 TiB 66.53 1.02 99 up
12 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 20 MiB 22 GiB
4.2 TiB 67.13 1.03 99 up
18 hdd 12.73340 1.00000 13 TiB 9.9 TiB 9.9 TiB 24 MiB 31 GiB
2.8 TiB 77.81 1.19 112 up
19 hdd 12.73340 1.00000 13 TiB 8.8 TiB 8.8 TiB 34 MiB 22 GiB
3.9 TiB 69.17 1.06 104 up
21 hdd 12.73340 1.00000 13 TiB 9.5 TiB 9.4 TiB 15 MiB 28 GiB
3.3 TiB 74.37 1.14 111 up
31 hdd 10.91409 1.00000 11 TiB 6.3 TiB 6.3 TiB 56 MiB 14 GiB
4.6 TiB 57.53 0.88 77 up
32 hdd 10.91409 1.00000 11 TiB 8.0 TiB 8.0 TiB 44 MiB 18 GiB
2.9 TiB 73.15 1.12 99 up
37 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.1 TiB 4.2 MiB 18 GiB
4.6 TiB 64.09 0.98 94 up
40 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.3 TiB 1.4 MiB 16 GiB
5.5 TiB 57.10 0.88 86 up
44 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 2.6 MiB 17 GiB
5.0 TiB 60.55 0.93 92 up
1 hdd 10.91409 1.00000 11 TiB 5.9 TiB 5.9 TiB 55 MiB 15 GiB
5.0 TiB 54.14 0.83 72 up
9 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.3 TiB 705 KiB 24 GiB
3.4 TiB 73.57 1.13 109 up
13 hdd 12.73340 1.00000 13 TiB 8.0 TiB 8.0 TiB 22 MiB 21 GiB
4.7 TiB 62.74 0.96 91 up
17 hdd 10.91409 1.00000 11 TiB 7.5 TiB 7.5 TiB 24 MiB 19 GiB
3.4 TiB 68.88 1.06 88 up
27 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.8 TiB 4.8 MiB 17 GiB
5.9 TiB 53.81 0.83 78 up
28 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 19 MiB 20 GiB
5.0 TiB 60.46 0.93 91 up
29 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 17 MiB 21 GiB
4.4 TiB 65.56 1.01 100 up
33 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.6 MiB 15 GiB
4.4 TiB 59.23 0.91 77 up
34 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.6 TiB 44 MiB 15 GiB
4.3 TiB 60.87 0.93 81 up
35 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.5 TiB 18 MiB 15 GiB
4.4 TiB 60.08 0.92 81 up
38 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 18 MiB 19 GiB
4.3 TiB 66.25 1.02 105 up
42 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.9 TiB 21 MiB 16 GiB
5.8 TiB 54.26 0.83 84 up
45 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 38 MiB 18 GiB
5.0 TiB 60.61 0.93 94 up
TOTAL 582 TiB 380 TiB 379 TiB 942 MiB 950 GiB
203 TiB 65.20
Thank you,
Ricardo
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
