|
Upmap is awesome. I ran it on our new cluster before we started ingesting data, so that the PG count is balanced on all OSDs. After ingesting about 315TB, it's still beautifully balanced. Note: we have a few nodes with
8TB OSDs, and the rest on 10TBs.
# ceph osd df plain
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144
1 mf1hdd 7.27739 1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144
2 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
3 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.37 1.02 144
4 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
5 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141
6 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142
7 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
8 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141
9 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
10 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
11 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.99 1.00 142
12 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141
13 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
14 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141
15 mf1hdd 7.27739 1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144
224 nvmemeta 0.02179 1.00000 22.3GiB 1.52GiB 20.8GiB 6.82 0.24 185
225 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.68 0.24 182
144 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
145 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
146 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
147 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
148 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
149 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
150 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175
151 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
152 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.87 1.00 173
153 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.16 1.01 175
154 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173
155 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
156 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
157 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
158 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174
159 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
242 nvmemeta 0.02179 1.00000 22.3GiB 1.50GiB 20.8GiB 6.70 0.24 182
243 nvmemeta 0.02179 1.00000 22.4GiB 1.45GiB 20.9GiB 6.48 0.23 183
160 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
161 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
162 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
163 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
164 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
165 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
166 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
167 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
168 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
169 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173
170 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175
171 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
172 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
173 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
174 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
175 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
244 nvmemeta 0.02179 1.00000 22.3GiB 1.49GiB 20.9GiB 6.65 0.24 182
245 nvmemeta 0.02179 1.00000 22.4GiB 1.53GiB 20.8GiB 6.86 0.25 185
176 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
177 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
178 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
179 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
180 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
181 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
182 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
183 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.94 1.00 174
184 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
185 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
186 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
187 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
188 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
189 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
190 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
191 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
246 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.64 0.24 184
247 nvmemeta 0.02179 1.00000 22.4GiB 1.47GiB 20.9GiB 6.57 0.24 183
192 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144
193 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.96 1.00 142
194 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.22TiB 28.31 1.01 144
195 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143
196 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
197 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143
198 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.22TiB 28.34 1.01 144
199 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142
200 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143
201 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144
202 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
203 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
204 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144
205 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141
206 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.25TiB 27.79 1.00 141
207 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142
248 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182
249 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.66 0.24 182
208 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
209 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
210 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.74 0.99 141
211 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.13 1.01 143
212 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.93 1.00 142
213 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141
214 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143
215 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141
216 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
217 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
218 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143
219 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
220 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.22TiB 28.22 1.01 143
221 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
222 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143
223 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143
250 nvmemeta 0.02179 1.00000 22.3GiB 1.46GiB 20.9GiB 6.54 0.23 183
251 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.66 0.24 183
16 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144
17 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143
18 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
19 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141
20 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143
21 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.17 1.01 143
22 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141
23 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
24 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
25 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141
26 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142
27 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
28 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
29 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
30 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142
31 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
226 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182
227 nvmemeta 0.02179 1.00000 22.4GiB 1.46GiB 20.9GiB 6.51 0.23 182
32 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
33 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.42TiB 28.00 1.00 174
34 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173
35 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
36 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
37 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
38 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
39 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
40 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
41 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
42 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
43 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173
44 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
45 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
46 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173
47 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
228 nvmemeta 0.02179 1.00000 22.3GiB 1.49GiB 20.9GiB 6.65 0.24 182
229 nvmemeta 0.02179 1.00000 22.4GiB 1.45GiB 20.9GiB 6.47 0.23 183
48 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.17 1.01 175
49 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
50 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
51 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
52 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
53 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
54 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
55 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
56 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
57 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173
58 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
59 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
60 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
61 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
62 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
63 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
230 nvmemeta 0.02179 1.00000 22.3GiB 1.45GiB 20.9GiB 6.48 0.23 183
231 nvmemeta 0.02179 1.00000 22.4GiB 1.48GiB 20.9GiB 6.63 0.24 182
64 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175
65 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
66 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
67 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
68 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
69 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
70 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
71 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
72 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
73 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
74 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
75 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174
76 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
77 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
78 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
79 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
232 nvmemeta 0.02179 1.00000 22.3GiB 1.47GiB 20.9GiB 6.57 0.24 182
233 nvmemeta 0.02179 1.00000 22.4GiB 1.50GiB 20.9GiB 6.70 0.24 184
80 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
81 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
82 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
83 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
84 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173
85 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
86 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
87 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
88 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
89 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
90 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
91 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
92 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
93 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
94 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175
95 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.78 0.99 173
234 nvmemeta 0.02179 1.00000 22.3GiB 1.51GiB 20.8GiB 6.77 0.24 185
235 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.65 0.24 182
96 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
97 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
98 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
99 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
100 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
101 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
102 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
103 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.21 1.01 175
104 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
105 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
106 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
107 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
108 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
109 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
110 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
111 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
236 nvmemeta 0.02179 1.00000 22.3GiB 1.46GiB 20.9GiB 6.53 0.23 185
237 nvmemeta 0.02179 1.00000 22.4GiB 1.46GiB 20.9GiB 6.54 0.23 182
112 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
113 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
114 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
115 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
116 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
117 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
118 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
119 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.96 1.00 174
120 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
121 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
122 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
123 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174
124 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
125 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
126 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
127 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
238 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182
239 nvmemeta 0.02179 1.00000 22.4GiB 1.51GiB 20.8GiB 6.75 0.24 182
128 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
129 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
130 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
131 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
132 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
133 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
134 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
135 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.13 1.01 175
136 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
137 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
138 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
139 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
140 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
141 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
142 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174
143 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
240 nvmemeta 0.02179 1.00000 22.3GiB 1.61GiB 20.7GiB 7.22 0.26 184
241 nvmemeta 0.02179 1.00000 22.4GiB 1.43GiB 20.9GiB 6.41 0.23 182
TOTAL 1.85PiB 528TiB 1.33PiB 27.93
MIN/MAX VAR: 0.23/1.02 STDDEV: 7.10
From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx>
Sent: Sunday, 21 October 2018 6:57:49 AM To: Janne Johansson Cc: ceph-users@xxxxxxxxxxxxxx; Peter Wienemann Subject: Re: ceph df space usage confusion - balancing needed? Ok, I'll try out the balancer end of the upcoming week then (after we've fixed a HW-issue with one of our mons
and the cooling system). Until then, any further advice and whether upmap is recommended over crush-compat (all clients are Luminous) are welcome ;-). Cheers, Oliver Am 20.10.18 um 21:26 schrieb Janne Johansson: > Ok, can't say "why" then, I'd reweigh them somewhat to even it out, > 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for > the MGRs, a script or just a few manual tweaks might be in order. > > Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth > <freyermuth@xxxxxxxxxxxxxxxxxx>: >> >> All OSDs are of the very same size. One OSD host has slightly more disks (33 instead of 31), though. >> So also that that can't explain the hefty difference. >> >> I attach the output of "ceph osd tree" and "ceph osd df". >> >> The crush rule for the ceph_data pool is: >> rule cephfs_data { >> id 2 >> type erasure >> min_size 3 >> max_size 6 >> step set_chooseleaf_tries 5 >> step set_choose_tries 100 >> step take default class hdd >> step chooseleaf indep 0 type host >> step emit >> } >> So that only considers the hdd device class. EC is done with k=4 m=2. >> >> So I don't see any imbalance on the hardware level, but only a somewhat uneven distribution of PGs. >> Am I missing something, or is this really just a case for the ceph balancer plugin? >> I'm just a bit astonished this effect is so huge. >> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even distribution without balancing? >> But it yields about 100 PGs per OSD, as you can see... >> >> -------------------------------------------------------------- >> # ceph osd tree >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 826.26428 root default >> -3 0.43700 host mon001 >> 0 ssd 0.21799 osd.0 up 1.00000 1.00000 >> 1 ssd 0.21799 osd.1 up 1.00000 1.00000 >> -5 0.43700 host mon002 >> 2 ssd 0.21799 osd.2 up 1.00000 1.00000 >> 3 ssd 0.21799 osd.3 up 1.00000 1.00000 >> -31 1.81898 host mon003 >> 230 ssd 0.90999 osd.230 up 1.00000 1.00000 >> 231 ssd 0.90999 osd.231 up 1.00000 1.00000 >> -10 116.64600 host osd001 >> 4 hdd 3.64499 osd.4 up 1.00000 1.00000 >> 5 hdd 3.64499 osd.5 up 1.00000 1.00000 >> 6 hdd 3.64499 osd.6 up 1.00000 1.00000 >> 7 hdd 3.64499 osd.7 up 1.00000 1.00000 >> 8 hdd 3.64499 osd.8 up 1.00000 1.00000 >> 9 hdd 3.64499 osd.9 up 1.00000 1.00000 >> 10 hdd 3.64499 osd.10 up 1.00000 1.00000 >> 11 hdd 3.64499 osd.11 up 1.00000 1.00000 >> 12 hdd 3.64499 osd.12 up 1.00000 1.00000 >> 13 hdd 3.64499 osd.13 up 1.00000 1.00000 >> 14 hdd 3.64499 osd.14 up 1.00000 1.00000 >> 15 hdd 3.64499 osd.15 up 1.00000 1.00000 >> 16 hdd 3.64499 osd.16 up 1.00000 1.00000 >> 17 hdd 3.64499 osd.17 up 1.00000 1.00000 >> 18 hdd 3.64499 osd.18 up 1.00000 1.00000 >> 19 hdd 3.64499 osd.19 up 1.00000 1.00000 >> 20 hdd 3.64499 osd.20 up 1.00000 1.00000 >> 21 hdd 3.64499 osd.21 up 1.00000 1.00000 >> 22 hdd 3.64499 osd.22 up 1.00000 1.00000 >> 23 hdd 3.64499 osd.23 up 1.00000 1.00000 >> 24 hdd 3.64499 osd.24 up 1.00000 1.00000 >> 25 hdd 3.64499 osd.25 up 1.00000 1.00000 >> 26 hdd 3.64499 osd.26 up 1.00000 1.00000 >> 27 hdd 3.64499 osd.27 up 1.00000 1.00000 >> 28 hdd 3.64499 osd.28 up 1.00000 1.00000 >> 29 hdd 3.64499 osd.29 up 1.00000 1.00000 >> 30 hdd 3.64499 osd.30 up 1.00000 1.00000 >> 31 hdd 3.64499 osd.31 up 1.00000 1.00000 >> 32 hdd 3.64499 osd.32 up 1.00000 1.00000 >> 33 hdd 3.64499 osd.33 up 1.00000 1.00000 >> 34 hdd 3.64499 osd.34 up 1.00000 1.00000 >> 35 hdd 3.64499 osd.35 up 1.00000 1.00000 >> -13 116.64600 host osd002 >> 36 hdd 3.64499 osd.36 up 1.00000 1.00000 >> 37 hdd 3.64499 osd.37 up 1.00000 1.00000 >> 38 hdd 3.64499 osd.38 up 1.00000 1.00000 >> 39 hdd 3.64499 osd.39 up 1.00000 1.00000 >> 40 hdd 3.64499 osd.40 up 1.00000 1.00000 >> 41 hdd 3.64499 osd.41 up 1.00000 1.00000 >> 42 hdd 3.64499 osd.42 up 1.00000 1.00000 >> 43 hdd 3.64499 osd.43 up 1.00000 1.00000 >> 44 hdd 3.64499 osd.44 up 1.00000 1.00000 >> 45 hdd 3.64499 osd.45 up 1.00000 1.00000 >> 46 hdd 3.64499 osd.46 up 1.00000 1.00000 >> 47 hdd 3.64499 osd.47 up 1.00000 1.00000 >> 48 hdd 3.64499 osd.48 up 1.00000 1.00000 >> 49 hdd 3.64499 osd.49 up 1.00000 1.00000 >> 50 hdd 3.64499 osd.50 up 1.00000 1.00000 >> 51 hdd 3.64499 osd.51 up 1.00000 1.00000 >> 52 hdd 3.64499 osd.52 up 1.00000 1.00000 >> 53 hdd 3.64499 osd.53 up 1.00000 1.00000 >> 54 hdd 3.64499 osd.54 up 1.00000 1.00000 >> 55 hdd 3.64499 osd.55 up 1.00000 1.00000 >> 56 hdd 3.64499 osd.56 up 1.00000 1.00000 >> 57 hdd 3.64499 osd.57 up 1.00000 1.00000 >> 58 hdd 3.64499 osd.58 up 1.00000 1.00000 >> 59 hdd 3.64499 osd.59 up 1.00000 1.00000 >> 60 hdd 3.64499 osd.60 up 1.00000 1.00000 >> 61 hdd 3.64499 osd.61 up 1.00000 1.00000 >> 62 hdd 3.64499 osd.62 up 1.00000 1.00000 >> 63 hdd 3.64499 osd.63 up 1.00000 1.00000 >> 64 hdd 3.64499 osd.64 up 1.00000 1.00000 >> 65 hdd 3.64499 osd.65 up 1.00000 1.00000 >> 66 hdd 3.64499 osd.66 up 1.00000 1.00000 >> 67 hdd 3.64499 osd.67 up 1.00000 1.00000 >> -16 116.63899 host osd003 >> 68 hdd 3.64499 osd.68 up 1.00000 1.00000 >> 69 hdd 3.64499 osd.69 up 1.00000 1.00000 >> 70 hdd 3.64499 osd.70 up 1.00000 1.00000 >> 71 hdd 3.64499 osd.71 up 1.00000 1.00000 >> 72 hdd 3.64499 osd.72 up 1.00000 1.00000 >> 73 hdd 3.64499 osd.73 up 1.00000 1.00000 >> 74 hdd 3.64499 osd.74 up 1.00000 1.00000 >> 75 hdd 3.64499 osd.75 up 1.00000 1.00000 >> 76 hdd 3.64499 osd.76 up 1.00000 1.00000 >> 77 hdd 3.64499 osd.77 up 1.00000 1.00000 >> 78 hdd 3.64499 osd.78 up 1.00000 1.00000 >> 79 hdd 3.64499 osd.79 up 1.00000 1.00000 >> 80 hdd 3.63899 osd.80 up 1.00000 1.00000 >> 81 hdd 3.64499 osd.81 up 1.00000 1.00000 >> 82 hdd 3.64499 osd.82 up 1.00000 1.00000 >> 83 hdd 3.64499 osd.83 up 1.00000 1.00000 >> 84 hdd 3.64499 osd.84 up 1.00000 1.00000 >> 85 hdd 3.64499 osd.85 up 1.00000 1.00000 >> 86 hdd 3.64499 osd.86 up 1.00000 1.00000 >> 87 hdd 3.64499 osd.87 up 1.00000 1.00000 >> 88 hdd 3.64499 osd.88 up 1.00000 1.00000 >> 89 hdd 3.64499 osd.89 up 1.00000 1.00000 >> 90 hdd 3.64499 osd.90 up 1.00000 1.00000 >> 91 hdd 3.64499 osd.91 up 1.00000 1.00000 >> 92 hdd 3.64499 osd.92 up 1.00000 1.00000 >> 93 hdd 3.64499 osd.93 up 1.00000 1.00000 >> 94 hdd 3.64499 osd.94 up 1.00000 1.00000 >> 95 hdd 3.64499 osd.95 up 1.00000 1.00000 >> 96 hdd 3.64499 osd.96 up 1.00000 1.00000 >> 97 hdd 3.64499 osd.97 up 1.00000 1.00000 >> 98 hdd 3.64499 osd.98 up 1.00000 1.00000 >> 99 hdd 3.64499 osd.99 up 1.00000 1.00000 >> -19 116.64600 host osd004 >> 100 hdd 3.64499 osd.100 up 1.00000 1.00000 >> 101 hdd 3.64499 osd.101 up 1.00000 1.00000 >> 102 hdd 3.64499 osd.102 up 1.00000 1.00000 >> 103 hdd 3.64499 osd.103 up 1.00000 1.00000 >> 104 hdd 3.64499 osd.104 up 1.00000 1.00000 >> 105 hdd 3.64499 osd.105 up 1.00000 1.00000 >> 106 hdd 3.64499 osd.106 up 1.00000 1.00000 >> 107 hdd 3.64499 osd.107 up 1.00000 1.00000 >> 108 hdd 3.64499 osd.108 up 1.00000 1.00000 >> 109 hdd 3.64499 osd.109 up 1.00000 1.00000 >> 110 hdd 3.64499 osd.110 up 1.00000 1.00000 >> 111 hdd 3.64499 osd.111 up 1.00000 1.00000 >> 112 hdd 3.64499 osd.112 up 1.00000 1.00000 >> 113 hdd 3.64499 osd.113 up 1.00000 1.00000 >> 114 hdd 3.64499 osd.114 up 1.00000 1.00000 >> 115 hdd 3.64499 osd.115 up 1.00000 1.00000 >> 116 hdd 3.64499 osd.116 up 1.00000 1.00000 >> 117 hdd 3.64499 osd.117 up 1.00000 1.00000 >> 118 hdd 3.64499 osd.118 up 1.00000 1.00000 >> 119 hdd 3.64499 osd.119 up 1.00000 1.00000 >> 120 hdd 3.64499 osd.120 up 1.00000 1.00000 >> 121 hdd 3.64499 osd.121 up 1.00000 1.00000 >> 122 hdd 3.64499 osd.122 up 1.00000 1.00000 >> 123 hdd 3.64499 osd.123 up 1.00000 1.00000 >> 124 hdd 3.64499 osd.124 up 1.00000 1.00000 >> 125 hdd 3.64499 osd.125 up 1.00000 1.00000 >> 126 hdd 3.64499 osd.126 up 1.00000 1.00000 >> 127 hdd 3.64499 osd.127 up 1.00000 1.00000 >> 128 hdd 3.64499 osd.128 up 1.00000 1.00000 >> 129 hdd 3.64499 osd.129 up 1.00000 1.00000 >> 130 hdd 3.64499 osd.130 up 1.00000 1.00000 >> 131 hdd 3.64499 osd.131 up 1.00000 1.00000 >> -22 116.64600 host osd005 >> 132 hdd 3.64499 osd.132 up 1.00000 1.00000 >> 133 hdd 3.64499 osd.133 up 1.00000 1.00000 >> 134 hdd 3.64499 osd.134 up 1.00000 1.00000 >> 135 hdd 3.64499 osd.135 up 1.00000 1.00000 >> 136 hdd 3.64499 osd.136 up 1.00000 1.00000 >> 137 hdd 3.64499 osd.137 up 1.00000 1.00000 >> 138 hdd 3.64499 osd.138 up 1.00000 1.00000 >> 139 hdd 3.64499 osd.139 up 1.00000 1.00000 >> 140 hdd 3.64499 osd.140 up 1.00000 1.00000 >> 141 hdd 3.64499 osd.141 up 1.00000 1.00000 >> 142 hdd 3.64499 osd.142 up 1.00000 1.00000 >> 143 hdd 3.64499 osd.143 up 1.00000 1.00000 >> 144 hdd 3.64499 osd.144 up 1.00000 1.00000 >> 145 hdd 3.64499 osd.145 up 1.00000 1.00000 >> 146 hdd 3.64499 osd.146 up 1.00000 1.00000 >> 147 hdd 3.64499 osd.147 up 1.00000 1.00000 >> 148 hdd 3.64499 osd.148 up 1.00000 1.00000 >> 149 hdd 3.64499 osd.149 up 1.00000 1.00000 >> 150 hdd 3.64499 osd.150 up 1.00000 1.00000 >> 151 hdd 3.64499 osd.151 up 1.00000 1.00000 >> 152 hdd 3.64499 osd.152 up 1.00000 1.00000 >> 153 hdd 3.64499 osd.153 up 1.00000 1.00000 >> 154 hdd 3.64499 osd.154 up 1.00000 1.00000 >> 155 hdd 3.64499 osd.155 up 1.00000 1.00000 >> 156 hdd 3.64499 osd.156 up 1.00000 1.00000 >> 157 hdd 3.64499 osd.157 up 1.00000 1.00000 >> 158 hdd 3.64499 osd.158 up 1.00000 1.00000 >> 159 hdd 3.64499 osd.159 up 1.00000 1.00000 >> 160 hdd 3.64499 osd.160 up 1.00000 1.00000 >> 161 hdd 3.64499 osd.161 up 1.00000 1.00000 >> 162 hdd 3.64499 osd.162 up 1.00000 1.00000 >> 163 hdd 3.64499 osd.163 up 1.00000 1.00000 >> -25 116.63335 host osd006 >> 164 hdd 3.64499 osd.164 up 1.00000 1.00000 >> 165 hdd 3.64499 osd.165 up 1.00000 1.00000 >> 166 hdd 3.64499 osd.166 up 1.00000 1.00000 >> 167 hdd 3.64499 osd.167 up 1.00000 1.00000 >> 168 hdd 3.64499 osd.168 up 1.00000 1.00000 >> 169 hdd 3.64499 osd.169 up 1.00000 1.00000 >> 170 hdd 3.64499 osd.170 up 1.00000 1.00000 >> 171 hdd 3.64499 osd.171 up 1.00000 1.00000 >> 172 hdd 3.64499 osd.172 up 1.00000 1.00000 >> 173 hdd 3.64499 osd.173 up 1.00000 1.00000 >> 174 hdd 3.64499 osd.174 up 1.00000 1.00000 >> 175 hdd 3.64499 osd.175 up 1.00000 1.00000 >> 176 hdd 3.64499 osd.176 up 1.00000 1.00000 >> 177 hdd 3.64499 osd.177 up 1.00000 1.00000 >> 178 hdd 3.64499 osd.178 up 1.00000 1.00000 >> 179 hdd 3.64499 osd.179 up 1.00000 1.00000 >> 180 hdd 3.64499 osd.180 up 1.00000 1.00000 >> 181 hdd 3.64499 osd.181 up 1.00000 1.00000 >> 182 hdd 3.64499 osd.182 up 1.00000 1.00000 >> 183 hdd 3.64499 osd.183 up 1.00000 1.00000 >> 184 hdd 3.64499 osd.184 up 1.00000 1.00000 >> 185 hdd 3.64499 osd.185 up 1.00000 1.00000 >> 186 hdd 3.64499 osd.186 up 1.00000 1.00000 >> 187 hdd 3.64499 osd.187 up 1.00000 1.00000 >> 188 hdd 3.64499 osd.188 up 1.00000 1.00000 >> 189 hdd 3.64499 osd.189 up 1.00000 1.00000 >> 190 hdd 3.64499 osd.190 up 1.00000 1.00000 >> 191 hdd 3.64499 osd.191 up 1.00000 1.00000 >> 192 hdd 3.63869 osd.192 up 1.00000 1.00000 >> 193 hdd 3.64499 osd.193 up 1.00000 1.00000 >> 194 hdd 3.64499 osd.194 up 1.00000 1.00000 >> 195 hdd 3.64499 osd.195 up 1.00000 1.00000 >> -28 123.71500 host osd007 >> 196 hdd 3.63899 osd.196 up 1.00000 1.00000 >> 197 hdd 3.63899 osd.197 up 1.00000 1.00000 >> 198 hdd 3.63899 osd.198 up 1.00000 1.00000 >> 199 hdd 3.63899 osd.199 up 1.00000 1.00000 >> 200 hdd 3.63899 osd.200 up 1.00000 1.00000 >> 201 hdd 3.63899 osd.201 up 1.00000 1.00000 >> 202 hdd 3.63899 osd.202 up 1.00000 1.00000 >> 203 hdd 3.63899 osd.203 up 1.00000 1.00000 >> 204 hdd 3.63899 osd.204 up 1.00000 1.00000 >> 205 hdd 3.63899 osd.205 up 1.00000 1.00000 >> 206 hdd 3.63899 osd.206 up 1.00000 1.00000 >> 207 hdd 3.63899 osd.207 up 1.00000 1.00000 >> 208 hdd 3.63899 osd.208 up 1.00000 1.00000 >> 209 hdd 3.63899 osd.209 up 1.00000 1.00000 >> 210 hdd 3.63899 osd.210 up 1.00000 1.00000 >> 211 hdd 3.63899 osd.211 up 1.00000 1.00000 >> 212 hdd 3.63899 osd.212 up 1.00000 1.00000 >> 213 hdd 3.63899 osd.213 up 1.00000 1.00000 >> 214 hdd 3.63899 osd.214 up 1.00000 1.00000 >> 215 hdd 3.63899 osd.215 up 1.00000 1.00000 >> 216 hdd 3.63899 osd.216 up 1.00000 1.00000 >> 217 hdd 3.63899 osd.217 up 1.00000 1.00000 >> 218 hdd 3.63899 osd.218 up 1.00000 1.00000 >> 219 hdd 3.63899 osd.219 up 1.00000 1.00000 >> 220 hdd 3.63899 osd.220 up 1.00000 1.00000 >> 221 hdd 3.63899 osd.221 up 1.00000 1.00000 >> 222 hdd 3.63899 osd.222 up 1.00000 1.00000 >> 223 hdd 3.63899 osd.223 up 1.00000 1.00000 >> 224 hdd 3.63899 osd.224 up 1.00000 1.00000 >> 225 hdd 3.63899 osd.225 up 1.00000 1.00000 >> 226 hdd 3.63899 osd.226 up 1.00000 1.00000 >> 227 hdd 3.63899 osd.227 up 1.00000 1.00000 >> 228 hdd 3.63899 osd.228 up 1.00000 1.00000 >> 229 hdd 3.63899 osd.229 up 1.00000 1.00000 >> >> # ceph osd df >> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS >> 0 ssd 0.21799 1.00000 223G 4305M 219G 1.88 0.04 64 >> 1 ssd 0.21799 1.00000 223G 3353M 220G 1.46 0.03 64 >> 2 ssd 0.21799 1.00000 223G 1836M 221G 0.80 0.02 56 >> 3 ssd 0.21799 1.00000 223G 2276M 221G 0.99 0.02 72 >> 230 ssd 0.90999 1.00000 931G 2775M 928G 0.29 0.01 60 >> 231 ssd 0.90999 1.00000 931G 3279M 928G 0.34 0.01 68 >> 4 hdd 3.64499 1.00000 3726G 1916G 1809G 51.45 1.02 111 >> 5 hdd 3.64499 1.00000 3726G 2144G 1581G 57.54 1.15 124 >> 6 hdd 3.64499 1.00000 3726G 1887G 1838G 50.66 1.01 109 >> 7 hdd 3.64499 1.00000 3726G 1642G 2083G 44.09 0.88 95 >> 8 hdd 3.64499 1.00000 3726G 1969G 1756G 52.86 1.05 114 >> 9 hdd 3.64499 1.00000 3726G 1577G 2148G 42.34 0.84 91 >> 10 hdd 3.64499 1.00000 3726G 2043G 1682G 54.85 1.09 118 >> 11 hdd 3.64499 1.00000 3726G 1957G 1768G 52.54 1.05 113 >> 12 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114 >> 13 hdd 3.64499 1.00000 3726G 1676G 2049G 45.00 0.90 97 >> 14 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102 >> 15 hdd 3.64499 1.00000 3726G 2161G 1564G 58.02 1.15 125 >> 16 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101 >> 17 hdd 3.64499 1.00000 3726G 2022G 1703G 54.29 1.08 117 >> 18 hdd 3.64499 1.00000 3726G 1784G 1941G 47.89 0.95 103 >> 19 hdd 3.64499 1.00000 3726G 1783G 1942G 47.85 0.95 103 >> 20 hdd 3.64499 1.00000 3726G 1731G 1994G 46.47 0.92 100 >> 21 hdd 3.64499 1.00000 3726G 1939G 1786G 52.06 1.04 112 >> 22 hdd 3.64499 1.00000 3726G 2061G 1664G 55.34 1.10 119 >> 23 hdd 3.64499 1.00000 3726G 1538G 2187G 41.29 0.82 89 >> 24 hdd 3.64499 1.00000 3726G 2168G 1557G 58.19 1.16 125 >> 25 hdd 3.64499 1.00000 3726G 2023G 1702G 54.32 1.08 117 >> 26 hdd 3.64499 1.00000 3726G 1900G 1825G 51.00 1.02 110 >> 27 hdd 3.64499 1.00000 3726G 2126G 1599G 57.08 1.14 123 >> 28 hdd 3.64499 1.00000 3726G 1938G 1787G 52.02 1.04 112 >> 29 hdd 3.64499 1.00000 3726G 1663G 2062G 44.65 0.89 96 >> 30 hdd 3.64499 1.00000 3726G 2163G 1562G 58.07 1.16 125 >> 31 hdd 3.64499 1.00000 3726G 1919G 1806G 51.52 1.03 111 >> 32 hdd 3.64499 1.00000 3726G 1696G 2029G 45.52 0.91 98 >> 33 hdd 3.64499 1.00000 3726G 1438G 2287G 38.61 0.77 83 >> 34 hdd 3.64499 1.00000 3726G 2073G 1652G 55.64 1.11 120 >> 35 hdd 3.64499 1.00000 3726G 2079G 1646G 55.81 1.11 120 >> 36 hdd 3.64499 1.00000 3726G 1955G 1770G 52.50 1.04 113 >> 37 hdd 3.64499 1.00000 3726G 2180G 1545G 58.51 1.16 126 >> 38 hdd 3.64499 1.00000 3726G 1664G 2061G 44.66 0.89 96 >> 39 hdd 3.64499 1.00000 3726G 2003G 1722G 53.77 1.07 116 >> 40 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103 >> 41 hdd 3.64499 1.00000 3726G 1903G 1822G 51.08 1.02 110 >> 42 hdd 3.64499 1.00000 3726G 1747G 1978G 46.89 0.93 101 >> 43 hdd 3.64499 1.00000 3726G 2021G 1704G 54.26 1.08 117 >> 44 hdd 3.64499 1.00000 3726G 1852G 1873G 49.71 0.99 107 >> 45 hdd 3.64499 1.00000 3726G 1522G 2203G 40.87 0.81 88 >> 46 hdd 3.64499 1.00000 3726G 1851G 1874G 49.69 0.99 107 >> 47 hdd 3.64499 1.00000 3726G 1698G 2027G 45.59 0.91 98 >> 48 hdd 3.64499 1.00000 3726G 1940G 1785G 52.08 1.04 112 >> 49 hdd 3.64499 1.00000 3726G 1746G 1979G 46.88 0.93 101 >> 50 hdd 3.64499 1.00000 3726G 1728G 1997G 46.40 0.92 100 >> 51 hdd 3.64499 1.00000 3726G 2093G 1632G 56.18 1.12 121 >> 52 hdd 3.64499 1.00000 3726G 2235G 1490G 60.01 1.19 129 >> 53 hdd 3.64499 1.00000 3726G 1871G 1854G 50.23 1.00 108 >> 54 hdd 3.64499 1.00000 3726G 2094G 1631G 56.22 1.12 121 >> 55 hdd 3.64499 1.00000 3726G 2422G 1303G 65.02 1.29 140 >> 56 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 >> 57 hdd 3.64499 1.00000 3726G 1729G 1996G 46.43 0.92 100 >> 58 hdd 3.64499 1.00000 3726G 1976G 1749G 53.05 1.06 114 >> 59 hdd 3.64499 1.00000 3726G 1954G 1771G 52.46 1.04 113 >> 60 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107 >> 61 hdd 3.64499 1.00000 3726G 2042G 1683G 54.82 1.09 118 >> 62 hdd 3.64499 1.00000 3726G 1921G 1804G 51.58 1.03 111 >> 63 hdd 3.64499 1.00000 3726G 2128G 1597G 57.13 1.14 123 >> 64 hdd 3.64499 1.00000 3726G 2037G 1688G 54.67 1.09 118 >> 65 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114 >> 66 hdd 3.64499 1.00000 3726G 1539G 2186G 41.32 0.82 89 >> 67 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 >> 68 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 >> 69 hdd 3.64499 1.00000 3726G 2092G 1633G 56.16 1.12 121 >> 70 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104 >> 71 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129 >> 72 hdd 3.64499 1.00000 3726G 1557G 2168G 41.79 0.83 90 >> 73 hdd 3.64499 1.00000 3726G 2300G 1425G 61.75 1.23 133 >> 74 hdd 3.64499 1.00000 3726G 2132G 1593G 57.24 1.14 123 >> 75 hdd 3.64499 1.00000 3726G 1818G 1907G 48.80 0.97 105 >> 76 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129 >> 77 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101 >> 78 hdd 3.64499 1.00000 3726G 1766G 1959G 47.41 0.94 102 >> 79 hdd 3.64499 1.00000 3726G 1678G 2047G 45.04 0.90 97 >> 80 hdd 3.63899 1.00000 3726G 1744G 1981G 46.82 0.93 102 >> 81 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 >> 82 hdd 3.64499 1.00000 3726G 1711G 2014G 45.93 0.91 99 >> 83 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100 >> 84 hdd 3.64499 1.00000 3726G 1834G 1891G 49.22 0.98 106 >> 85 hdd 3.64499 1.00000 3726G 1709G 2016G 45.88 0.91 99 >> 86 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112 >> 87 hdd 3.64499 1.00000 3726G 2131G 1594G 57.19 1.14 123 >> 88 hdd 3.64499 1.00000 3726G 1559G 2166G 41.85 0.83 90 >> 89 hdd 3.64499 1.00000 3726G 1645G 2080G 44.16 0.88 95 >> 90 hdd 3.64499 1.00000 3726G 2045G 1680G 54.89 1.09 118 >> 91 hdd 3.64499 1.00000 3726G 2164G 1561G 58.09 1.16 125 >> 92 hdd 3.64499 1.00000 3726G 1922G 1803G 51.59 1.03 111 >> 93 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112 >> 94 hdd 3.64499 1.00000 3726G 1971G 1754G 52.91 1.05 114 >> 95 hdd 3.64499 1.00000 3726G 1917G 1808G 51.47 1.02 111 >> 96 hdd 3.64499 1.00000 3726G 1814G 1911G 48.70 0.97 105 >> 97 hdd 3.64499 1.00000 3726G 1660G 2065G 44.57 0.89 96 >> 98 hdd 3.64499 1.00000 3726G 1904G 1821G 51.10 1.02 110 >> 99 hdd 3.64499 1.00000 3726G 2112G 1613G 56.70 1.13 122 >> 100 hdd 3.64499 1.00000 3726G 1869G 1856G 50.18 1.00 108 >> 101 hdd 3.64499 1.00000 3726G 2005G 1720G 53.84 1.07 116 >> 102 hdd 3.64499 1.00000 3726G 2039G 1686G 54.73 1.09 118 >> 103 hdd 3.64499 1.00000 3726G 2093G 1632G 56.20 1.12 121 >> 104 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 >> 105 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 >> 106 hdd 3.64499 1.00000 3726G 1956G 1769G 52.51 1.05 113 >> 107 hdd 3.64499 1.00000 3726G 2077G 1648G 55.74 1.11 120 >> 108 hdd 3.64499 1.00000 3726G 1990G 1735G 53.43 1.06 115 >> 109 hdd 3.64499 1.00000 3726G 1781G 1944G 47.80 0.95 103 >> 110 hdd 3.64499 1.00000 3726G 1869G 1856G 50.17 1.00 108 >> 111 hdd 3.64499 1.00000 3726G 2023G 1702G 54.30 1.08 117 >> 112 hdd 3.64499 1.00000 3726G 2007G 1718G 53.89 1.07 116 >> 113 hdd 3.64499 1.00000 3726G 1851G 1874G 49.70 0.99 107 >> 114 hdd 3.64499 1.00000 3726G 1832G 1893G 49.18 0.98 106 >> 115 hdd 3.64499 1.00000 3726G 2098G 1627G 56.32 1.12 121 >> 116 hdd 3.64499 1.00000 3726G 1866G 1859G 50.10 1.00 108 >> 117 hdd 3.64499 1.00000 3726G 1764G 1961G 47.36 0.94 102 >> 118 hdd 3.64499 1.00000 3726G 1816G 1909G 48.76 0.97 105 >> 119 hdd 3.64499 1.00000 3726G 1733G 1992G 46.53 0.93 100 >> 120 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 >> 121 hdd 3.64499 1.00000 3726G 1657G 2068G 44.48 0.89 96 >> 122 hdd 3.64499 1.00000 3726G 1715G 2010G 46.04 0.92 99 >> 123 hdd 3.64499 1.00000 3726G 1974G 1751G 53.00 1.05 114 >> 124 hdd 3.64499 1.00000 3726G 1731G 1994G 46.48 0.93 100 >> 125 hdd 3.64499 1.00000 3726G 1897G 1828G 50.94 1.01 110 >> 126 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115 >> 127 hdd 3.64499 1.00000 3726G 1972G 1753G 52.93 1.05 114 >> 128 hdd 3.64499 1.00000 3726G 1888G 1837G 50.68 1.01 109 >> 129 hdd 3.64499 1.00000 3726G 2025G 1700G 54.35 1.08 117 >> 130 hdd 3.64499 1.00000 3726G 1747G 1978G 46.90 0.93 101 >> 131 hdd 3.64499 1.00000 3726G 2061G 1664G 55.32 1.10 119 >> 132 hdd 3.64499 1.00000 3726G 2185G 1540G 58.65 1.17 126 >> 133 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102 >> 134 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103 >> 135 hdd 3.64499 1.00000 3726G 1675G 2050G 44.96 0.89 97 >> 136 hdd 3.64499 1.00000 3726G 1889G 1836G 50.70 1.01 109 >> 137 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113 >> 138 hdd 3.64499 1.00000 3726G 1694G 2031G 45.49 0.91 98 >> 139 hdd 3.64499 1.00000 3726G 2114G 1611G 56.76 1.13 122 >> 140 hdd 3.64499 1.00000 3726G 1726G 1999G 46.35 0.92 100 >> 141 hdd 3.64499 1.00000 3726G 1988G 1737G 53.36 1.06 115 >> 142 hdd 3.64499 1.00000 3726G 2181G 1544G 58.55 1.17 126 >> 143 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107 >> 144 hdd 3.64499 1.00000 3726G 2144G 1581G 57.56 1.15 124 >> 145 hdd 3.64499 1.00000 3726G 1868G 1857G 50.14 1.00 108 >> 146 hdd 3.64499 1.00000 3726G 2078G 1647G 55.77 1.11 120 >> 147 hdd 3.64499 1.00000 3726G 1992G 1733G 53.49 1.06 115 >> 148 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100 >> 149 hdd 3.64499 1.00000 3726G 2162G 1563G 58.03 1.15 125 >> 150 hdd 3.64499 1.00000 3726G 2005G 1720G 53.83 1.07 116 >> 151 hdd 3.64499 1.00000 3726G 1918G 1807G 51.49 1.02 111 >> 152 hdd 3.64499 1.00000 3726G 1921G 1804G 51.57 1.03 111 >> 153 hdd 3.64499 1.00000 3726G 1783G 1942G 47.86 0.95 103 >> 154 hdd 3.64499 1.00000 3726G 2005G 1720G 53.82 1.07 116 >> 155 hdd 3.64499 1.00000 3726G 1853G 1873G 49.73 0.99 107 >> 156 hdd 3.64499 1.00000 3726G 1626G 2099G 43.65 0.87 94 >> 157 hdd 3.64499 1.00000 3726G 1802G 1923G 48.38 0.96 104 >> 158 hdd 3.64499 1.00000 3726G 1765G 1960G 47.39 0.94 102 >> 159 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115 >> 160 hdd 3.64499 1.00000 3726G 1901G 1824G 51.02 1.02 110 >> 161 hdd 3.64499 1.00000 3726G 1906G 1819G 51.16 1.02 110 >> 162 hdd 3.64499 1.00000 3726G 1763G 1962G 47.33 0.94 102 >> 163 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104 >> 164 hdd 3.64499 1.00000 3726G 1922G 1803G 51.61 1.03 111 >> 165 hdd 3.64499 1.00000 3726G 1731G 1994G 46.46 0.92 100 >> 166 hdd 3.64499 1.00000 3726G 2027G 1699G 54.40 1.08 117 >> 167 hdd 3.64499 1.00000 3726G 1956G 1769G 52.50 1.04 113 >> 168 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107 >> 169 hdd 3.64499 1.00000 3726G 1884G 1841G 50.59 1.01 109 >> 170 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 >> 171 hdd 3.64499 1.00000 3726G 1898G 1827G 50.96 1.01 110 >> 172 hdd 3.64499 1.00000 3726G 1781G 1944G 47.82 0.95 103 >> 173 hdd 3.64499 1.00000 3726G 1628G 2097G 43.71 0.87 94 >> 174 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 >> 175 hdd 3.64499 1.00000 3726G 1905G 1820G 51.15 1.02 110 >> 176 hdd 3.64499 1.00000 3726G 1709G 2016G 45.89 0.91 99 >> 177 hdd 3.64499 1.00000 3726G 1991G 1734G 53.44 1.06 115 >> 178 hdd 3.64499 1.00000 3726G 1973G 1752G 52.97 1.05 114 >> 179 hdd 3.64499 1.00000 3726G 1574G 2151G 42.26 0.84 91 >> 180 hdd 3.64499 1.00000 3726G 1917G 1808G 51.46 1.02 111 >> 181 hdd 3.64499 1.00000 3726G 1937G 1788G 52.00 1.03 112 >> 182 hdd 3.64499 1.00000 3726G 1989G 1736G 53.40 1.06 115 >> 183 hdd 3.64499 1.00000 3726G 2039G 1686G 54.74 1.09 118 >> 184 hdd 3.64499 1.00000 3726G 1868G 1857G 50.16 1.00 108 >> 185 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113 >> 186 hdd 3.64499 1.00000 3726G 1850G 1875G 49.66 0.99 107 >> 187 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107 >> 188 hdd 3.64499 1.00000 3726G 1814G 1911G 48.69 0.97 105 >> 189 hdd 3.64499 1.00000 3726G 2286G 1439G 61.36 1.22 132 >> 190 hdd 3.64499 1.00000 3726G 2060G 1665G 55.31 1.10 119 >> 191 hdd 3.64499 1.00000 3726G 1836G 1889G 49.29 0.98 106 >> 192 hdd 3.63869 1.00000 3726G 1721G 2004G 46.21 0.92 101 >> 193 hdd 3.64499 1.00000 3726G 1953G 1772G 52.43 1.04 113 >> 194 hdd 3.64499 1.00000 3726G 1541G 2184G 41.37 0.82 89 >> 195 hdd 3.64499 1.00000 3726G 1837G 1888G 49.32 0.98 106 >> 196 hdd 3.63899 1.00000 3726G 1705G 2020G 45.79 0.91 100 >> 197 hdd 3.63899 1.00000 3726G 1743G 1982G 46.78 0.93 102 >> 198 hdd 3.63899 1.00000 3726G 1982G 1743G 53.21 1.06 116 >> 199 hdd 3.63899 1.00000 3726G 1382G 2343G 37.11 0.74 81 >> 200 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111 >> 201 hdd 3.63899 1.00000 3726G 1845G 1880G 49.54 0.99 108 >> 202 hdd 3.63899 1.00000 3726G 1658G 2067G 44.51 0.89 97 >> 203 hdd 3.63899 1.00000 3726G 2066G 1659G 55.47 1.10 121 >> 204 hdd 3.63899 1.00000 3726G 1672G 2053G 44.90 0.89 98 >> 205 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89 >> 206 hdd 3.63899 1.00000 3726G 1796G 1929G 48.22 0.96 105 >> 207 hdd 3.63899 1.00000 3726G 1555G 2170G 41.75 0.83 91 >> 208 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102 >> 209 hdd 3.63899 1.00000 3726G 2016G 1709G 54.11 1.08 118 >> 210 hdd 3.63899 1.00000 3726G 2136G 1589G 57.34 1.14 125 >> 211 hdd 3.63899 1.00000 3726G 1656G 2069G 44.46 0.88 97 >> 212 hdd 3.63899 1.00000 3726G 1553G 2172G 41.71 0.83 91 >> 213 hdd 3.63899 1.00000 3726G 1758G 1967G 47.19 0.94 103 >> 214 hdd 3.63899 1.00000 3726G 1603G 2122G 43.03 0.86 94 >> 215 hdd 3.63899 1.00000 3726G 1809G 1916G 48.57 0.97 106 >> 216 hdd 3.63899 1.00000 3726G 1760G 1965G 47.26 0.94 103 >> 217 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102 >> 218 hdd 3.63899 1.00000 3726G 1880G 1845G 50.46 1.00 110 >> 219 hdd 3.63899 1.00000 3726G 1693G 2032G 45.45 0.90 99 >> 220 hdd 3.63899 1.00000 3726G 2032G 1693G 54.56 1.09 119 >> 221 hdd 3.63899 1.00000 3726G 1842G 1883G 49.44 0.98 108 >> 222 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89 >> 223 hdd 3.63899 1.00000 3726G 1470G 2255G 39.46 0.79 86 >> 224 hdd 3.63899 1.00000 3726G 1808G 1917G 48.53 0.97 106 >> 225 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111 >> 226 hdd 3.63899 1.00000 3726G 1776G 1949G 47.69 0.95 104 >> 227 hdd 3.63899 1.00000 3726G 1929G 1796G 51.78 1.03 113 >> 228 hdd 3.63899 1.00000 3726G 1657G 2068G 44.48 0.89 97 >> 229 hdd 3.63899 1.00000 3726G 1843G 1882G 49.47 0.98 108 >> TOTAL 825T 414T 410T 50.24 >> MIN/MAX VAR: 0.01/1.29 STDDEV: 9.22 >> -------------------------------------------------------------- >> >> Am 20.10.18 um 20:35 schrieb Janne Johansson: >>> Yes, if you have uneven sizes I guess you could end up in a situation >>> where you have >>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces >>> the pool to have one >>> PG replica on the 1TB OSD, then it would be possible to state "this >>> pool cant write more than X G" >>> but when it is full, there would be free space left on some of the >>> 2TB-OSDs, but which the pool >>> cant utilize. Probably same for uneven OSD hosts if you have those. >>> >>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth >>> <freyermuth@xxxxxxxxxxxxxxxxxx>: >>>> >>>> Dear Janne, >>>> >>>> yes, of course. But since we only have two pools here, this can not explain the difference. >>>> The metadata is replicated (3 copies) across ssd drives, and we have < 3 TB of total raw storage for that. >>>> So looking at the raw space usage, we can ignore that. >>>> >>>> All the rest is used for the ceph_data pool. So the ceph_data pool, in terms of raw storage, is about 50 % used. >>>> >>>> But in terms of storage shown for that pool, it's almost 63 % %USED. >>>> So I guess this can purely be from bad balancing, correct? >>>> >>>> Cheers, >>>> Oliver >>>> >>>> Am 20.10.18 um 19:49 schrieb Janne Johansson: >>>>> Do mind that drives may have more than one pool on them, so RAW space >>>>> is what it says, how much free space there is. Then the avail and >>>>> %USED on per-pool stats will take replication into account, it can >>>>> tell how much data you may write into that particular pool, given that >>>>> pools replication or EC settings. >>>>> >>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth >>>>> <freyermuth@xxxxxxxxxxxxxxxxxx>: >>>>>> >>>>>> Dear Cephalopodians, >>>>>> >>>>>> as many others, I'm also a bit confused by "ceph df" output >>>>>> in a pretty straightforward configuration. >>>>>> >>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile. >>>>>> >>>>>> I get: >>>>>> ---------------------------------------------------------------------------- >>>>>> # ceph df >>>>>> GLOBAL: >>>>>> SIZE AVAIL RAW USED %RAW USED >>>>>> 824T 410T 414T 50.26 >>>>>> POOLS: >>>>>> NAME ID USED %USED MAX AVAIL OBJECTS >>>>>> cephfs_metadata 1 452M 0.05 860G 365774 >>>>>> cephfs_data 2 275T 62.68 164T 75056403 >>>>>> ---------------------------------------------------------------------------- >>>>>> >>>>>> So about 50 % of raw space are used, but already ~63 % of filesystem space are used. >>>>>> Is this purely from imperfect balancing? >>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % usage down to 37.12 %. >>>>>> >>>>>> We did not yet use the balancer plugin. >>>>>> We don't have any pre-luminous clients. >>>>>> In that setup, I take it that "upmap" mode would be recommended - correct? >>>>>> Any "gotchas" using that on luminous? >>>>>> >>>>>> Cheers, >>>>>> Oliver >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
