V14.2.4
So, this is not new, this happens every time there is a rebalance, now because of raising PGs. PG balancer is disabled because I thought it was the reason but apparently it's not, but it ain't helping either.
Ceph is totally borged, it's only moving data on nearfull OSDs causing issues. See this after PG raise.
health: HEALTH_WARN
3 nearfull osd(s)
2 pool(s) nearfull
3 nearfull osd(s)
2 pool(s) nearfull
08:44 am
92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB 1.6 TiB 82.79 1.25 39 up
71 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 88 KiB 20 GiB 1.4 TiB 84.09 1.27 38 up
21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB 1.5 TiB 84.05 1.27 36 up
08:54 am
92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB 1.6 TiB 82.81 1.25 39 up
71 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.6 TiB 88 KiB 20 GiB 1.4 TiB 84.14 1.27 38 up
21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB 1.4 TiB 84.10 1.27 36 up
14 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 17 GiB 2.6 TiB 71.33 1.07 32 up
19 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB 2.8 TiB 68.81 1.04 30 up
22 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 92 KiB 17 GiB 2.8 TiB 68.90 1.04 32 up
25 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 219 KiB 17 GiB 2.9 TiB 68.11 1.03 31 up
30 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 20 KiB 17 GiB 2.6 TiB 71.41 1.08 33 up
33 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 40 KiB 17 GiB 2.6 TiB 71.30 1.07 32 up
34 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 17 GiB 2.6 TiB 71.33 1.07 30 up
35 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 124 KiB 17 GiB 2.5 TiB 72.61 1.09 32 up
12 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 18 GiB 2.4 TiB 73.84 1.11 32 up
16 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 96 KiB 17 GiB 2.6 TiB 71.08 1.07 29 up
17 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 60 KiB 17 GiB 2.6 TiB 71.41 1.08 31 up
20 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 92 KiB 17 GiB 2.9 TiB 68.57 1.03 28 up
23 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 17 GiB 2.6 TiB 71.37 1.08 29 up
26 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 84 KiB 17 GiB 2.7 TiB 70.02 1.06 30 up
28 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 28 KiB 17 GiB 2.7 TiB 70.11 1.06 30 up
31 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 56 KiB 17 GiB 2.6 TiB 71.26 1.07 32 up
13 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 18 GiB 2.4 TiB 73.84 1.11 31 up
15 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB 2.6 TiB 71.35 1.08 29 up
18 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 76 KiB 16 GiB 3.3 TiB 63.70 0.96 26 up
21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB 1.4 TiB 84.10 1.27 36 up
24 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 64 KiB 15 GiB 3.3 TiB 63.67 0.96 28 up
27 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 17 GiB 3.1 TiB 66.03 1.00 28 up
29 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.3 TiB 28 KiB 18 GiB 2.7 TiB 69.93 1.05 34 up
32 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 20 KiB 17 GiB 3.1 TiB 66.20 1.00 28 up
37 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 32 KiB 18 GiB 2.7 TiB 70.59 1.06 31 up
39 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 32 KiB 19 GiB 2.7 TiB 70.50 1.06 29 up
41 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB 2.8 TiB 68.79 1.04 30 up
43 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 48 KiB 17 GiB 2.8 TiB 68.84 1.04 28 up
45 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 80 KiB 18 GiB 2.4 TiB 74.02 1.12 33 up
46 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 36 KiB 18 GiB 2.4 TiB 73.88 1.11 30 up
48 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 101 KiB 17 GiB 2.5 TiB 72.57 1.09 31 up
50 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 96 KiB 17 GiB 2.8 TiB 68.86 1.04 31 up
36 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 60 KiB 18 GiB 2.4 TiB 73.82 1.11 32 up
38 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 76 KiB 17 GiB 2.7 TiB 70.62 1.06 33 up
40 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 8 KiB 18 GiB 2.4 TiB 73.89 1.11 33 up
42 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 120 KiB 17 GiB 2.8 TiB 68.82 1.04 28 up
44 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 68 KiB 17 GiB 2.6 TiB 71.39 1.08 29 up
47 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB 2.7 TiB 70.06 1.06 29 up
49 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 40 KiB 16 GiB 3.1 TiB 66.25 1.00 28 up
51 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 60 KiB 15 GiB 3.3 TiB 63.73 0.96 27 up
52 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 200 KiB 17 GiB 2.6 TiB 71.09 1.07 33 up
53 hdd 9.09470 0.79999 9.1 TiB 7.2 TiB 7.2 TiB 48 KiB 19 GiB 1.9 TiB 79.03 1.19 33 up
54 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 44 KiB 15 GiB 3.5 TiB 61.14 0.92 25 up 55 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 36 KiB 15 GiB 3.3 TiB 63.72 0.96 26 up
56 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 49 KiB 18 GiB 2.4 TiB 73.91 1.11 31 up
57 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 72 KiB 17 GiB 2.8 TiB 68.88 1.04 29 up
58 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 20 KiB 18 GiB 2.6 TiB 71.40 1.08 30 up
59 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 60 KiB 16 GiB 3.0 TiB 67.51 1.02 29 up
60 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 32 KiB 16 GiB 3.1 TiB 66.14 1.00 26 up
61 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 32 KiB 17 GiB 2.9 TiB 68.59 1.03 32 up
62 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 80 KiB 17 GiB 2.7 TiB 70.06 1.06 29 up
63 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 76 KiB 18 GiB 2.4 TiB 73.85 1.11 33 up
64 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 44 KiB 17 GiB 2.8 TiB 68.80 1.04 31 up
65 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB 2.7 TiB 70.10 1.06 32 up
66 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB 2.8 TiB 68.77 1.04 30 up
67 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 112 KiB 17 GiB 2.6 TiB 71.36 1.08 31 up
84 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 60 KiB 17 GiB 2.8 TiB 68.84 1.04 30 up
85 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 48 KiB 19 GiB 2.1 TiB 76.46 1.15 32 up
86 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 76 KiB 16 GiB 2.8 TiB 68.80 1.04 29 up
87 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB 3.1 TiB 66.28 1.00 27 up
88 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 28 KiB 17 GiB 3.1 TiB 65.51 0.99 27 up
89 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 116 KiB 17 GiB 2.8 TiB 68.90 1.04 28 up
90 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 56 KiB 16 GiB 3.1 TiB 66.29 1.00 28 up
91 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 16 KiB 16 GiB 3.1 TiB 66.24 1.00 30 up
76 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 76 KiB 17 GiB 2.8 TiB 68.78 1.04 27 up
77 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB 2.6 TiB 71.39 1.08 29 up
78 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 40 KiB 17 GiB 2.6 TiB 71.38 1.08 32 up
79 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 56 KiB 17 GiB 2.6 TiB 71.29 1.07 33 up
80 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 18 GiB 2.6 TiB 71.36 1.08 30 up
81 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB 2.6 TiB 71.36 1.08 32 up
82 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 36 KiB 17 GiB 2.8 TiB 68.82 1.04 30 up
83 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 36 KiB 17 GiB 2.7 TiB 70.07 1.06 29 up
68 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 20 KiB 17 GiB 2.7 TiB 70.06 1.06 31 up
69 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 16 KiB 18 GiB 2.4 TiB 73.96 1.11 32 up
70 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 64 KiB 17 GiB 2.8 TiB 68.84 1.04 29 up
71 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.6 TiB 88 KiB 20 GiB 1.4 TiB 84.14 1.27 38 up
72 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 48 KiB 16 GiB 3.0 TiB 67.55 1.02 28 up
73 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 80 KiB 16 GiB 3.1 TiB 66.20 1.00 30 up
74 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 17 GiB 2.6 TiB 71.25 1.07 31 up
75 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 44 KiB 16 GiB 3.1 TiB 66.17 1.00 26 up
121 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 76 KiB 15 GiB 3.5 TiB 61.18 0.92 28 up
123 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 72 KiB 17 GiB 2.6 TiB 71.47 1.08 29 up
124 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 17 GiB 2.6 TiB 71.29 1.07 31 up
126 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 64 KiB 16 GiB 2.9 TiB 68.65 1.03 29 up
128 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB 17 GiB 2.5 TiB 72.65 1.09 32 up
130 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 96 KiB 16 GiB 3.1 TiB 66.33 1.00 28 up
132 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 84 KiB 17 GiB 2.4 TiB 73.99 1.11 31 up
134 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 48 KiB 18 GiB 2.4 TiB 73.22 1.10 30 up
112 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 64 KiB 18 GiB 2.4 TiB 73.86 1.11 31 up
113 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB 15 GiB 3.3 TiB 63.77 0.96 28 up
114 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 16 GiB 3.1 TiB 66.29 1.00 28 up
115 hdd 9.09470 0.89999 9.1 TiB 6.7 TiB 6.7 TiB 40 KiB 18 GiB 2.4 TiB 73.70 1.11 29 up
116 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 4 KiB 17 GiB 2.6 TiB 71.44 1.08 29 up 117 hdd 9.09470 0.89999 9.1 TiB 6.7 TiB 6.7 TiB 120 KiB 18 GiB 2.4 TiB 73.90 1.11 29 up
118 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 16 KiB 15 GiB 3.4 TiB 62.88 0.95 25 up 119 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 108 KiB 16 GiB 2.8 TiB 68.83 1.04 30 up
120 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 68 KiB 16 GiB 3.1 TiB 65.48 0.99 29 up
122 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 100 KiB 17 GiB 2.5 TiB 72.63 1.09 31 up
125 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 64 KiB 17 GiB 2.4 TiB 73.85 1.11 32 up
127 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 171 KiB 17 GiB 2.5 TiB 72.58 1.09 34 up
129 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 48 KiB 17 GiB 2.6 TiB 71.39 1.08 30 up
131 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 68 KiB 17 GiB 2.6 TiB 71.38 1.08 29 up
133 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 28 KiB 17 GiB 2.4 TiB 73.97 1.11 31 up
135 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB 17 GiB 2.5 TiB 72.58 1.09 30 up
92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB 1.6 TiB 82.81 1.25 39 up
93 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 88 KiB 16 GiB 2.8 TiB 68.83 1.04 30 up
94 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 32 KiB 13 GiB 4.2 TiB 53.51 0.81 25 up
95 hdd 8.00000 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB 3.1 TiB 66.28 1.00 27 up
96 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 76 KiB 15 GiB 3.3 TiB 63.72 0.96 27 up
97 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 68 KiB 15 GiB 3.3 TiB 63.61 0.96 27 up
98 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 17 GiB 2.4 TiB 73.82 1.11 29 up
99 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 76 KiB 17 GiB 2.6 TiB 71.09 1.07 33 up I really need some help with this, it's becoming unbearable to work with, I can't attend 24/7 to manually rebalance, this is not how its supposed to be. This is a Nautilus only issue
Thanks
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
