Dear Cephalopodians,
thanks for all your feedback!
I finally "pushed the button" and let upmap run for ~36 hours.
Previously, we had ~63 % usage of our CephFS with only 50 % raw usage, now, we see only 53.77 % usage.
That's as close as I expect things to ever become, and we gained about 70 TiB of free storage by that, which is almost one file server.
So the outcome is really close to perfection :-).
I'm leaving the balancer active now in upmap mode. Any bad experiences with leaving it active "forever"?
Cheers and many thanks again,
Oliver
Am 23.10.18 um 01:14 schrieb Linh Vu:
> Upmap is awesome. I ran it on our new cluster before we started ingesting data, so that the PG count is balanced on all OSDs. After ingesting about 315TB, it's still beautifully balanced. Note: we have a few nodes with 8TB OSDs, and the rest on 10TBs.
>
>
> # ceph osd df plain
> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
> 0 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144
> 1 mf1hdd 7.27739 1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144
> 2 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 3 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.37 1.02 144
> 4 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 5 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141
> 6 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142
> 7 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
> 8 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141
> 9 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
> 10 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
> 11 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.99 1.00 142
> 12 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141
> 13 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 14 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141
> 15 mf1hdd 7.27739 1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144
> 224 nvmemeta 0.02179 1.00000 22.3GiB 1.52GiB 20.8GiB 6.82 0.24 185
> 225 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.68 0.24 182
> 144 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 145 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 146 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 147 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 148 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 149 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 150 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175
> 151 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 152 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.87 1.00 173
> 153 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.16 1.01 175
> 154 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173
> 155 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 156 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 157 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 158 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174
> 159 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 242 nvmemeta 0.02179 1.00000 22.3GiB 1.50GiB 20.8GiB 6.70 0.24 182
> 243 nvmemeta 0.02179 1.00000 22.4GiB 1.45GiB 20.9GiB 6.48 0.23 183
> 160 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
> 161 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 162 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 163 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 164 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 165 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 166 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 167 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 168 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 169 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173
> 170 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175
> 171 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 172 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 173 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 174 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 175 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
> 244 nvmemeta 0.02179 1.00000 22.3GiB 1.49GiB 20.9GiB 6.65 0.24 182
> 245 nvmemeta 0.02179 1.00000 22.4GiB 1.53GiB 20.8GiB 6.86 0.25 185
> 176 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 177 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 178 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 179 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 180 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 181 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 182 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 183 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.94 1.00 174
> 184 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 185 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 186 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 187 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 188 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 189 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 190 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 191 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
> 246 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.64 0.24 184
> 247 nvmemeta 0.02179 1.00000 22.4GiB 1.47GiB 20.9GiB 6.57 0.24 183
> 192 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144
> 193 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.96 1.00 142
> 194 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.22TiB 28.31 1.01 144
> 195 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143
> 196 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
> 197 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143
> 198 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.22TiB 28.34 1.01 144
> 199 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142
> 200 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143
> 201 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144
> 202 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
> 203 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
> 204 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144
> 205 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141
> 206 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.25TiB 27.79 1.00 141
> 207 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142
> 248 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182
> 249 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.66 0.24 182
> 208 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 209 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
> 210 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.74 0.99 141
> 211 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.13 1.01 143
> 212 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.93 1.00 142
> 213 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141
> 214 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143
> 215 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141
> 216 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
> 217 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 218 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143
> 219 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 220 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.22TiB 28.22 1.01 143
> 221 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
> 222 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143
> 223 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143
> 250 nvmemeta 0.02179 1.00000 22.3GiB 1.46GiB 20.9GiB 6.54 0.23 183
> 251 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.66 0.24 183
> 16 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144
> 17 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143
> 18 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
> 19 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141
> 20 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143
> 21 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.17 1.01 143
> 22 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141
> 23 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 24 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
> 25 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141
> 26 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142
> 27 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142
> 28 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142
> 29 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142
> 30 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142
> 31 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144
> 226 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182
> 227 nvmemeta 0.02179 1.00000 22.4GiB 1.46GiB 20.9GiB 6.51 0.23 182
> 32 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 33 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.42TiB 28.00 1.00 174
> 34 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173
> 35 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 36 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 37 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
> 38 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
> 39 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 40 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 41 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 42 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 43 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173
> 44 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
> 45 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 46 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173
> 47 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 228 nvmemeta 0.02179 1.00000 22.3GiB 1.49GiB 20.9GiB 6.65 0.24 182
> 229 nvmemeta 0.02179 1.00000 22.4GiB 1.45GiB 20.9GiB 6.47 0.23 183
> 48 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.17 1.01 175
> 49 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 50 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 51 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 52 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 53 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 54 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 55 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 56 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 57 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173
> 58 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 59 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 60 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 61 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
> 62 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 63 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173
> 230 nvmemeta 0.02179 1.00000 22.3GiB 1.45GiB 20.9GiB 6.48 0.23 183
> 231 nvmemeta 0.02179 1.00000 22.4GiB 1.48GiB 20.9GiB 6.63 0.24 182
> 64 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175
> 65 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 66 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 67 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 68 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 69 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 70 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
> 71 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 72 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 73 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 74 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 75 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174
> 76 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 77 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 78 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 79 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 232 nvmemeta 0.02179 1.00000 22.3GiB 1.47GiB 20.9GiB 6.57 0.24 182
> 233 nvmemeta 0.02179 1.00000 22.4GiB 1.50GiB 20.9GiB 6.70 0.24 184
> 80 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 81 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 82 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 83 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 84 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173
> 85 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 86 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 87 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 88 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 89 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 90 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 91 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 92 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 93 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 94 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175
> 95 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.78 0.99 173
> 234 nvmemeta 0.02179 1.00000 22.3GiB 1.51GiB 20.8GiB 6.77 0.24 185
> 235 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.65 0.24 182
> 96 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 97 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 98 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 99 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
> 100 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 101 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 102 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 103 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.21 1.01 175
> 104 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 105 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 106 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 107 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 108 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 109 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 110 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 111 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 236 nvmemeta 0.02179 1.00000 22.3GiB 1.46GiB 20.9GiB 6.53 0.23 185
> 237 nvmemeta 0.02179 1.00000 22.4GiB 1.46GiB 20.9GiB 6.54 0.23 182
> 112 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 113 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 114 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174
> 115 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 116 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 117 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 118 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 119 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.96 1.00 174
> 120 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 121 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 122 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173
> 123 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174
> 124 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 125 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174
> 126 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174
> 127 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 238 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182
> 239 nvmemeta 0.02179 1.00000 22.4GiB 1.51GiB 20.8GiB 6.75 0.24 182
> 128 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173
> 129 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 130 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174
> 131 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 132 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 133 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 134 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 135 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.13 1.01 175
> 136 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174
> 137 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 138 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 139 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173
> 140 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173
> 141 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 142 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174
> 143 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173
> 240 nvmemeta 0.02179 1.00000 22.3GiB 1.61GiB 20.7GiB 7.22 0.26 184
> 241 nvmemeta 0.02179 1.00000 22.4GiB 1.43GiB 20.9GiB 6.41 0.23 182
> TOTAL 1.85PiB 528TiB 1.33PiB 27.93
> MIN/MAX VAR: 0.23/1.02 STDDEV: 7.10
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx>
> *Sent:* Sunday, 21 October 2018 6:57:49 AM
> *To:* Janne Johansson
> *Cc:* ceph-users@xxxxxxxxxxxxxx; Peter Wienemann
> *Subject:* Re: ceph df space usage confusion - balancing needed?
>
> Ok, I'll try out the balancer end of the upcoming week then (after we've fixed a HW-issue with one of our mons
> and the cooling system).
>
> Until then, any further advice and whether upmap is recommended over crush-compat (all clients are Luminous) are welcome ;-).
>
> Cheers,
> Oliver
>
> Am 20.10.18 um 21:26 schrieb Janne Johansson:
>> Ok, can't say "why" then, I'd reweigh them somewhat to even it out,
>> 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for
>> the MGRs, a script or just a few manual tweaks might be in order.
>>
>> Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth
>> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>>
>>> All OSDs are of the very same size. One OSD host has slightly more disks (33 instead of 31), though.
>>> So also that that can't explain the hefty difference.
>>>
>>> I attach the output of "ceph osd tree" and "ceph osd df".
>>>
>>> The crush rule for the ceph_data pool is:
>>> rule cephfs_data {
>>> id 2
>>> type erasure
>>> min_size 3
>>> max_size 6
>>> step set_chooseleaf_tries 5
>>> step set_choose_tries 100
>>> step take default class hdd
>>> step chooseleaf indep 0 type host
>>> step emit
>>> }
>>> So that only considers the hdd device class. EC is done with k=4 m=2.
>>>
>>> So I don't see any imbalance on the hardware level, but only a somewhat uneven distribution of PGs.
>>> Am I missing something, or is this really just a case for the ceph balancer plugin?
>>> I'm just a bit astonished this effect is so huge.
>>> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even distribution without balancing?
>>> But it yields about 100 PGs per OSD, as you can see...
>>>
>>> --------------------------------------------------------------
>>> # ceph osd tree
>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
>>> -1 826.26428 root default
>>> -3 0.43700 host mon001
>>> 0 ssd 0.21799 osd.0 up 1.00000 1.00000
>>> 1 ssd 0.21799 osd.1 up 1.00000 1.00000
>>> -5 0.43700 host mon002
>>> 2 ssd 0.21799 osd.2 up 1.00000 1.00000
>>> 3 ssd 0.21799 osd.3 up 1.00000 1.00000
>>> -31 1.81898 host mon003
>>> 230 ssd 0.90999 osd.230 up 1.00000 1.00000
>>> 231 ssd 0.90999 osd.231 up 1.00000 1.00000
>>> -10 116.64600 host osd001
>>> 4 hdd 3.64499 osd.4 up 1.00000 1.00000
>>> 5 hdd 3.64499 osd.5 up 1.00000 1.00000
>>> 6 hdd 3.64499 osd.6 up 1.00000 1.00000
>>> 7 hdd 3.64499 osd.7 up 1.00000 1.00000
>>> 8 hdd 3.64499 osd.8 up 1.00000 1.00000
>>> 9 hdd 3.64499 osd.9 up 1.00000 1.00000
>>> 10 hdd 3.64499 osd.10 up 1.00000 1.00000
>>> 11 hdd 3.64499 osd.11 up 1.00000 1.00000
>>> 12 hdd 3.64499 osd.12 up 1.00000 1.00000
>>> 13 hdd 3.64499 osd.13 up 1.00000 1.00000
>>> 14 hdd 3.64499 osd.14 up 1.00000 1.00000
>>> 15 hdd 3.64499 osd.15 up 1.00000 1.00000
>>> 16 hdd 3.64499 osd.16 up 1.00000 1.00000
>>> 17 hdd 3.64499 osd.17 up 1.00000 1.00000
>>> 18 hdd 3.64499 osd.18 up 1.00000 1.00000
>>> 19 hdd 3.64499 osd.19 up 1.00000 1.00000
>>> 20 hdd 3.64499 osd.20 up 1.00000 1.00000
>>> 21 hdd 3.64499 osd.21 up 1.00000 1.00000
>>> 22 hdd 3.64499 osd.22 up 1.00000 1.00000
>>> 23 hdd 3.64499 osd.23 up 1.00000 1.00000
>>> 24 hdd 3.64499 osd.24 up 1.00000 1.00000
>>> 25 hdd 3.64499 osd.25 up 1.00000 1.00000
>>> 26 hdd 3.64499 osd.26 up 1.00000 1.00000
>>> 27 hdd 3.64499 osd.27 up 1.00000 1.00000
>>> 28 hdd 3.64499 osd.28 up 1.00000 1.00000
>>> 29 hdd 3.64499 osd.29 up 1.00000 1.00000
>>> 30 hdd 3.64499 osd.30 up 1.00000 1.00000
>>> 31 hdd 3.64499 osd.31 up 1.00000 1.00000
>>> 32 hdd 3.64499 osd.32 up 1.00000 1.00000
>>> 33 hdd 3.64499 osd.33 up 1.00000 1.00000
>>> 34 hdd 3.64499 osd.34 up 1.00000 1.00000
>>> 35 hdd 3.64499 osd.35 up 1.00000 1.00000
>>> -13 116.64600 host osd002
>>> 36 hdd 3.64499 osd.36 up 1.00000 1.00000
>>> 37 hdd 3.64499 osd.37 up 1.00000 1.00000
>>> 38 hdd 3.64499 osd.38 up 1.00000 1.00000
>>> 39 hdd 3.64499 osd.39 up 1.00000 1.00000
>>> 40 hdd 3.64499 osd.40 up 1.00000 1.00000
>>> 41 hdd 3.64499 osd.41 up 1.00000 1.00000
>>> 42 hdd 3.64499 osd.42 up 1.00000 1.00000
>>> 43 hdd 3.64499 osd.43 up 1.00000 1.00000
>>> 44 hdd 3.64499 osd.44 up 1.00000 1.00000
>>> 45 hdd 3.64499 osd.45 up 1.00000 1.00000
>>> 46 hdd 3.64499 osd.46 up 1.00000 1.00000
>>> 47 hdd 3.64499 osd.47 up 1.00000 1.00000
>>> 48 hdd 3.64499 osd.48 up 1.00000 1.00000
>>> 49 hdd 3.64499 osd.49 up 1.00000 1.00000
>>> 50 hdd 3.64499 osd.50 up 1.00000 1.00000
>>> 51 hdd 3.64499 osd.51 up 1.00000 1.00000
>>> 52 hdd 3.64499 osd.52 up 1.00000 1.00000
>>> 53 hdd 3.64499 osd.53 up 1.00000 1.00000
>>> 54 hdd 3.64499 osd.54 up 1.00000 1.00000
>>> 55 hdd 3.64499 osd.55 up 1.00000 1.00000
>>> 56 hdd 3.64499 osd.56 up 1.00000 1.00000
>>> 57 hdd 3.64499 osd.57 up 1.00000 1.00000
>>> 58 hdd 3.64499 osd.58 up 1.00000 1.00000
>>> 59 hdd 3.64499 osd.59 up 1.00000 1.00000
>>> 60 hdd 3.64499 osd.60 up 1.00000 1.00000
>>> 61 hdd 3.64499 osd.61 up 1.00000 1.00000
>>> 62 hdd 3.64499 osd.62 up 1.00000 1.00000
>>> 63 hdd 3.64499 osd.63 up 1.00000 1.00000
>>> 64 hdd 3.64499 osd.64 up 1.00000 1.00000
>>> 65 hdd 3.64499 osd.65 up 1.00000 1.00000
>>> 66 hdd 3.64499 osd.66 up 1.00000 1.00000
>>> 67 hdd 3.64499 osd.67 up 1.00000 1.00000
>>> -16 116.63899 host osd003
>>> 68 hdd 3.64499 osd.68 up 1.00000 1.00000
>>> 69 hdd 3.64499 osd.69 up 1.00000 1.00000
>>> 70 hdd 3.64499 osd.70 up 1.00000 1.00000
>>> 71 hdd 3.64499 osd.71 up 1.00000 1.00000
>>> 72 hdd 3.64499 osd.72 up 1.00000 1.00000
>>> 73 hdd 3.64499 osd.73 up 1.00000 1.00000
>>> 74 hdd 3.64499 osd.74 up 1.00000 1.00000
>>> 75 hdd 3.64499 osd.75 up 1.00000 1.00000
>>> 76 hdd 3.64499 osd.76 up 1.00000 1.00000
>>> 77 hdd 3.64499 osd.77 up 1.00000 1.00000
>>> 78 hdd 3.64499 osd.78 up 1.00000 1.00000
>>> 79 hdd 3.64499 osd.79 up 1.00000 1.00000
>>> 80 hdd 3.63899 osd.80 up 1.00000 1.00000
>>> 81 hdd 3.64499 osd.81 up 1.00000 1.00000
>>> 82 hdd 3.64499 osd.82 up 1.00000 1.00000
>>> 83 hdd 3.64499 osd.83 up 1.00000 1.00000
>>> 84 hdd 3.64499 osd.84 up 1.00000 1.00000
>>> 85 hdd 3.64499 osd.85 up 1.00000 1.00000
>>> 86 hdd 3.64499 osd.86 up 1.00000 1.00000
>>> 87 hdd 3.64499 osd.87 up 1.00000 1.00000
>>> 88 hdd 3.64499 osd.88 up 1.00000 1.00000
>>> 89 hdd 3.64499 osd.89 up 1.00000 1.00000
>>> 90 hdd 3.64499 osd.90 up 1.00000 1.00000
>>> 91 hdd 3.64499 osd.91 up 1.00000 1.00000
>>> 92 hdd 3.64499 osd.92 up 1.00000 1.00000
>>> 93 hdd 3.64499 osd.93 up 1.00000 1.00000
>>> 94 hdd 3.64499 osd.94 up 1.00000 1.00000
>>> 95 hdd 3.64499 osd.95 up 1.00000 1.00000
>>> 96 hdd 3.64499 osd.96 up 1.00000 1.00000
>>> 97 hdd 3.64499 osd.97 up 1.00000 1.00000
>>> 98 hdd 3.64499 osd.98 up 1.00000 1.00000
>>> 99 hdd 3.64499 osd.99 up 1.00000 1.00000
>>> -19 116.64600 host osd004
>>> 100 hdd 3.64499 osd.100 up 1.00000 1.00000
>>> 101 hdd 3.64499 osd.101 up 1.00000 1.00000
>>> 102 hdd 3.64499 osd.102 up 1.00000 1.00000
>>> 103 hdd 3.64499 osd.103 up 1.00000 1.00000
>>> 104 hdd 3.64499 osd.104 up 1.00000 1.00000
>>> 105 hdd 3.64499 osd.105 up 1.00000 1.00000
>>> 106 hdd 3.64499 osd.106 up 1.00000 1.00000
>>> 107 hdd 3.64499 osd.107 up 1.00000 1.00000
>>> 108 hdd 3.64499 osd.108 up 1.00000 1.00000
>>> 109 hdd 3.64499 osd.109 up 1.00000 1.00000
>>> 110 hdd 3.64499 osd.110 up 1.00000 1.00000
>>> 111 hdd 3.64499 osd.111 up 1.00000 1.00000
>>> 112 hdd 3.64499 osd.112 up 1.00000 1.00000
>>> 113 hdd 3.64499 osd.113 up 1.00000 1.00000
>>> 114 hdd 3.64499 osd.114 up 1.00000 1.00000
>>> 115 hdd 3.64499 osd.115 up 1.00000 1.00000
>>> 116 hdd 3.64499 osd.116 up 1.00000 1.00000
>>> 117 hdd 3.64499 osd.117 up 1.00000 1.00000
>>> 118 hdd 3.64499 osd.118 up 1.00000 1.00000
>>> 119 hdd 3.64499 osd.119 up 1.00000 1.00000
>>> 120 hdd 3.64499 osd.120 up 1.00000 1.00000
>>> 121 hdd 3.64499 osd.121 up 1.00000 1.00000
>>> 122 hdd 3.64499 osd.122 up 1.00000 1.00000
>>> 123 hdd 3.64499 osd.123 up 1.00000 1.00000
>>> 124 hdd 3.64499 osd.124 up 1.00000 1.00000
>>> 125 hdd 3.64499 osd.125 up 1.00000 1.00000
>>> 126 hdd 3.64499 osd.126 up 1.00000 1.00000
>>> 127 hdd 3.64499 osd.127 up 1.00000 1.00000
>>> 128 hdd 3.64499 osd.128 up 1.00000 1.00000
>>> 129 hdd 3.64499 osd.129 up 1.00000 1.00000
>>> 130 hdd 3.64499 osd.130 up 1.00000 1.00000
>>> 131 hdd 3.64499 osd.131 up 1.00000 1.00000
>>> -22 116.64600 host osd005
>>> 132 hdd 3.64499 osd.132 up 1.00000 1.00000
>>> 133 hdd 3.64499 osd.133 up 1.00000 1.00000
>>> 134 hdd 3.64499 osd.134 up 1.00000 1.00000
>>> 135 hdd 3.64499 osd.135 up 1.00000 1.00000
>>> 136 hdd 3.64499 osd.136 up 1.00000 1.00000
>>> 137 hdd 3.64499 osd.137 up 1.00000 1.00000
>>> 138 hdd 3.64499 osd.138 up 1.00000 1.00000
>>> 139 hdd 3.64499 osd.139 up 1.00000 1.00000
>>> 140 hdd 3.64499 osd.140 up 1.00000 1.00000
>>> 141 hdd 3.64499 osd.141 up 1.00000 1.00000
>>> 142 hdd 3.64499 osd.142 up 1.00000 1.00000
>>> 143 hdd 3.64499 osd.143 up 1.00000 1.00000
>>> 144 hdd 3.64499 osd.144 up 1.00000 1.00000
>>> 145 hdd 3.64499 osd.145 up 1.00000 1.00000
>>> 146 hdd 3.64499 osd.146 up 1.00000 1.00000
>>> 147 hdd 3.64499 osd.147 up 1.00000 1.00000
>>> 148 hdd 3.64499 osd.148 up 1.00000 1.00000
>>> 149 hdd 3.64499 osd.149 up 1.00000 1.00000
>>> 150 hdd 3.64499 osd.150 up 1.00000 1.00000
>>> 151 hdd 3.64499 osd.151 up 1.00000 1.00000
>>> 152 hdd 3.64499 osd.152 up 1.00000 1.00000
>>> 153 hdd 3.64499 osd.153 up 1.00000 1.00000
>>> 154 hdd 3.64499 osd.154 up 1.00000 1.00000
>>> 155 hdd 3.64499 osd.155 up 1.00000 1.00000
>>> 156 hdd 3.64499 osd.156 up 1.00000 1.00000
>>> 157 hdd 3.64499 osd.157 up 1.00000 1.00000
>>> 158 hdd 3.64499 osd.158 up 1.00000 1.00000
>>> 159 hdd 3.64499 osd.159 up 1.00000 1.00000
>>> 160 hdd 3.64499 osd.160 up 1.00000 1.00000
>>> 161 hdd 3.64499 osd.161 up 1.00000 1.00000
>>> 162 hdd 3.64499 osd.162 up 1.00000 1.00000
>>> 163 hdd 3.64499 osd.163 up 1.00000 1.00000
>>> -25 116.63335 host osd006
>>> 164 hdd 3.64499 osd.164 up 1.00000 1.00000
>>> 165 hdd 3.64499 osd.165 up 1.00000 1.00000
>>> 166 hdd 3.64499 osd.166 up 1.00000 1.00000
>>> 167 hdd 3.64499 osd.167 up 1.00000 1.00000
>>> 168 hdd 3.64499 osd.168 up 1.00000 1.00000
>>> 169 hdd 3.64499 osd.169 up 1.00000 1.00000
>>> 170 hdd 3.64499 osd.170 up 1.00000 1.00000
>>> 171 hdd 3.64499 osd.171 up 1.00000 1.00000
>>> 172 hdd 3.64499 osd.172 up 1.00000 1.00000
>>> 173 hdd 3.64499 osd.173 up 1.00000 1.00000
>>> 174 hdd 3.64499 osd.174 up 1.00000 1.00000
>>> 175 hdd 3.64499 osd.175 up 1.00000 1.00000
>>> 176 hdd 3.64499 osd.176 up 1.00000 1.00000
>>> 177 hdd 3.64499 osd.177 up 1.00000 1.00000
>>> 178 hdd 3.64499 osd.178 up 1.00000 1.00000
>>> 179 hdd 3.64499 osd.179 up 1.00000 1.00000
>>> 180 hdd 3.64499 osd.180 up 1.00000 1.00000
>>> 181 hdd 3.64499 osd.181 up 1.00000 1.00000
>>> 182 hdd 3.64499 osd.182 up 1.00000 1.00000
>>> 183 hdd 3.64499 osd.183 up 1.00000 1.00000
>>> 184 hdd 3.64499 osd.184 up 1.00000 1.00000
>>> 185 hdd 3.64499 osd.185 up 1.00000 1.00000
>>> 186 hdd 3.64499 osd.186 up 1.00000 1.00000
>>> 187 hdd 3.64499 osd.187 up 1.00000 1.00000
>>> 188 hdd 3.64499 osd.188 up 1.00000 1.00000
>>> 189 hdd 3.64499 osd.189 up 1.00000 1.00000
>>> 190 hdd 3.64499 osd.190 up 1.00000 1.00000
>>> 191 hdd 3.64499 osd.191 up 1.00000 1.00000
>>> 192 hdd 3.63869 osd.192 up 1.00000 1.00000
>>> 193 hdd 3.64499 osd.193 up 1.00000 1.00000
>>> 194 hdd 3.64499 osd.194 up 1.00000 1.00000
>>> 195 hdd 3.64499 osd.195 up 1.00000 1.00000
>>> -28 123.71500 host osd007
>>> 196 hdd 3.63899 osd.196 up 1.00000 1.00000
>>> 197 hdd 3.63899 osd.197 up 1.00000 1.00000
>>> 198 hdd 3.63899 osd.198 up 1.00000 1.00000
>>> 199 hdd 3.63899 osd.199 up 1.00000 1.00000
>>> 200 hdd 3.63899 osd.200 up 1.00000 1.00000
>>> 201 hdd 3.63899 osd.201 up 1.00000 1.00000
>>> 202 hdd 3.63899 osd.202 up 1.00000 1.00000
>>> 203 hdd 3.63899 osd.203 up 1.00000 1.00000
>>> 204 hdd 3.63899 osd.204 up 1.00000 1.00000
>>> 205 hdd 3.63899 osd.205 up 1.00000 1.00000
>>> 206 hdd 3.63899 osd.206 up 1.00000 1.00000
>>> 207 hdd 3.63899 osd.207 up 1.00000 1.00000
>>> 208 hdd 3.63899 osd.208 up 1.00000 1.00000
>>> 209 hdd 3.63899 osd.209 up 1.00000 1.00000
>>> 210 hdd 3.63899 osd.210 up 1.00000 1.00000
>>> 211 hdd 3.63899 osd.211 up 1.00000 1.00000
>>> 212 hdd 3.63899 osd.212 up 1.00000 1.00000
>>> 213 hdd 3.63899 osd.213 up 1.00000 1.00000
>>> 214 hdd 3.63899 osd.214 up 1.00000 1.00000
>>> 215 hdd 3.63899 osd.215 up 1.00000 1.00000
>>> 216 hdd 3.63899 osd.216 up 1.00000 1.00000
>>> 217 hdd 3.63899 osd.217 up 1.00000 1.00000
>>> 218 hdd 3.63899 osd.218 up 1.00000 1.00000
>>> 219 hdd 3.63899 osd.219 up 1.00000 1.00000
>>> 220 hdd 3.63899 osd.220 up 1.00000 1.00000
>>> 221 hdd 3.63899 osd.221 up 1.00000 1.00000
>>> 222 hdd 3.63899 osd.222 up 1.00000 1.00000
>>> 223 hdd 3.63899 osd.223 up 1.00000 1.00000
>>> 224 hdd 3.63899 osd.224 up 1.00000 1.00000
>>> 225 hdd 3.63899 osd.225 up 1.00000 1.00000
>>> 226 hdd 3.63899 osd.226 up 1.00000 1.00000
>>> 227 hdd 3.63899 osd.227 up 1.00000 1.00000
>>> 228 hdd 3.63899 osd.228 up 1.00000 1.00000
>>> 229 hdd 3.63899 osd.229 up 1.00000 1.00000
>>>
>>> # ceph osd df
>>> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
>>> 0 ssd 0.21799 1.00000 223G 4305M 219G 1.88 0.04 64
>>> 1 ssd 0.21799 1.00000 223G 3353M 220G 1.46 0.03 64
>>> 2 ssd 0.21799 1.00000 223G 1836M 221G 0.80 0.02 56
>>> 3 ssd 0.21799 1.00000 223G 2276M 221G 0.99 0.02 72
>>> 230 ssd 0.90999 1.00000 931G 2775M 928G 0.29 0.01 60
>>> 231 ssd 0.90999 1.00000 931G 3279M 928G 0.34 0.01 68
>>> 4 hdd 3.64499 1.00000 3726G 1916G 1809G 51.45 1.02 111
>>> 5 hdd 3.64499 1.00000 3726G 2144G 1581G 57.54 1.15 124
>>> 6 hdd 3.64499 1.00000 3726G 1887G 1838G 50.66 1.01 109
>>> 7 hdd 3.64499 1.00000 3726G 1642G 2083G 44.09 0.88 95
>>> 8 hdd 3.64499 1.00000 3726G 1969G 1756G 52.86 1.05 114
>>> 9 hdd 3.64499 1.00000 3726G 1577G 2148G 42.34 0.84 91
>>> 10 hdd 3.64499 1.00000 3726G 2043G 1682G 54.85 1.09 118
>>> 11 hdd 3.64499 1.00000 3726G 1957G 1768G 52.54 1.05 113
>>> 12 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114
>>> 13 hdd 3.64499 1.00000 3726G 1676G 2049G 45.00 0.90 97
>>> 14 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102
>>> 15 hdd 3.64499 1.00000 3726G 2161G 1564G 58.02 1.15 125
>>> 16 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101
>>> 17 hdd 3.64499 1.00000 3726G 2022G 1703G 54.29 1.08 117
>>> 18 hdd 3.64499 1.00000 3726G 1784G 1941G 47.89 0.95 103
>>> 19 hdd 3.64499 1.00000 3726G 1783G 1942G 47.85 0.95 103
>>> 20 hdd 3.64499 1.00000 3726G 1731G 1994G 46.47 0.92 100
>>> 21 hdd 3.64499 1.00000 3726G 1939G 1786G 52.06 1.04 112
>>> 22 hdd 3.64499 1.00000 3726G 2061G 1664G 55.34 1.10 119
>>> 23 hdd 3.64499 1.00000 3726G 1538G 2187G 41.29 0.82 89
>>> 24 hdd 3.64499 1.00000 3726G 2168G 1557G 58.19 1.16 125
>>> 25 hdd 3.64499 1.00000 3726G 2023G 1702G 54.32 1.08 117
>>> 26 hdd 3.64499 1.00000 3726G 1900G 1825G 51.00 1.02 110
>>> 27 hdd 3.64499 1.00000 3726G 2126G 1599G 57.08 1.14 123
>>> 28 hdd 3.64499 1.00000 3726G 1938G 1787G 52.02 1.04 112
>>> 29 hdd 3.64499 1.00000 3726G 1663G 2062G 44.65 0.89 96
>>> 30 hdd 3.64499 1.00000 3726G 2163G 1562G 58.07 1.16 125
>>> 31 hdd 3.64499 1.00000 3726G 1919G 1806G 51.52 1.03 111
>>> 32 hdd 3.64499 1.00000 3726G 1696G 2029G 45.52 0.91 98
>>> 33 hdd 3.64499 1.00000 3726G 1438G 2287G 38.61 0.77 83
>>> 34 hdd 3.64499 1.00000 3726G 2073G 1652G 55.64 1.11 120
>>> 35 hdd 3.64499 1.00000 3726G 2079G 1646G 55.81 1.11 120
>>> 36 hdd 3.64499 1.00000 3726G 1955G 1770G 52.50 1.04 113
>>> 37 hdd 3.64499 1.00000 3726G 2180G 1545G 58.51 1.16 126
>>> 38 hdd 3.64499 1.00000 3726G 1664G 2061G 44.66 0.89 96
>>> 39 hdd 3.64499 1.00000 3726G 2003G 1722G 53.77 1.07 116
>>> 40 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103
>>> 41 hdd 3.64499 1.00000 3726G 1903G 1822G 51.08 1.02 110
>>> 42 hdd 3.64499 1.00000 3726G 1747G 1978G 46.89 0.93 101
>>> 43 hdd 3.64499 1.00000 3726G 2021G 1704G 54.26 1.08 117
>>> 44 hdd 3.64499 1.00000 3726G 1852G 1873G 49.71 0.99 107
>>> 45 hdd 3.64499 1.00000 3726G 1522G 2203G 40.87 0.81 88
>>> 46 hdd 3.64499 1.00000 3726G 1851G 1874G 49.69 0.99 107
>>> 47 hdd 3.64499 1.00000 3726G 1698G 2027G 45.59 0.91 98
>>> 48 hdd 3.64499 1.00000 3726G 1940G 1785G 52.08 1.04 112
>>> 49 hdd 3.64499 1.00000 3726G 1746G 1979G 46.88 0.93 101
>>> 50 hdd 3.64499 1.00000 3726G 1728G 1997G 46.40 0.92 100
>>> 51 hdd 3.64499 1.00000 3726G 2093G 1632G 56.18 1.12 121
>>> 52 hdd 3.64499 1.00000 3726G 2235G 1490G 60.01 1.19 129
>>> 53 hdd 3.64499 1.00000 3726G 1871G 1854G 50.23 1.00 108
>>> 54 hdd 3.64499 1.00000 3726G 2094G 1631G 56.22 1.12 121
>>> 55 hdd 3.64499 1.00000 3726G 2422G 1303G 65.02 1.29 140
>>> 56 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
>>> 57 hdd 3.64499 1.00000 3726G 1729G 1996G 46.43 0.92 100
>>> 58 hdd 3.64499 1.00000 3726G 1976G 1749G 53.05 1.06 114
>>> 59 hdd 3.64499 1.00000 3726G 1954G 1771G 52.46 1.04 113
>>> 60 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107
>>> 61 hdd 3.64499 1.00000 3726G 2042G 1683G 54.82 1.09 118
>>> 62 hdd 3.64499 1.00000 3726G 1921G 1804G 51.58 1.03 111
>>> 63 hdd 3.64499 1.00000 3726G 2128G 1597G 57.13 1.14 123
>>> 64 hdd 3.64499 1.00000 3726G 2037G 1688G 54.67 1.09 118
>>> 65 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114
>>> 66 hdd 3.64499 1.00000 3726G 1539G 2186G 41.32 0.82 89
>>> 67 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
>>> 68 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
>>> 69 hdd 3.64499 1.00000 3726G 2092G 1633G 56.16 1.12 121
>>> 70 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104
>>> 71 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129
>>> 72 hdd 3.64499 1.00000 3726G 1557G 2168G 41.79 0.83 90
>>> 73 hdd 3.64499 1.00000 3726G 2300G 1425G 61.75 1.23 133
>>> 74 hdd 3.64499 1.00000 3726G 2132G 1593G 57.24 1.14 123
>>> 75 hdd 3.64499 1.00000 3726G 1818G 1907G 48.80 0.97 105
>>> 76 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129
>>> 77 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101
>>> 78 hdd 3.64499 1.00000 3726G 1766G 1959G 47.41 0.94 102
>>> 79 hdd 3.64499 1.00000 3726G 1678G 2047G 45.04 0.90 97
>>> 80 hdd 3.63899 1.00000 3726G 1744G 1981G 46.82 0.93 102
>>> 81 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
>>> 82 hdd 3.64499 1.00000 3726G 1711G 2014G 45.93 0.91 99
>>> 83 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100
>>> 84 hdd 3.64499 1.00000 3726G 1834G 1891G 49.22 0.98 106
>>> 85 hdd 3.64499 1.00000 3726G 1709G 2016G 45.88 0.91 99
>>> 86 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112
>>> 87 hdd 3.64499 1.00000 3726G 2131G 1594G 57.19 1.14 123
>>> 88 hdd 3.64499 1.00000 3726G 1559G 2166G 41.85 0.83 90
>>> 89 hdd 3.64499 1.00000 3726G 1645G 2080G 44.16 0.88 95
>>> 90 hdd 3.64499 1.00000 3726G 2045G 1680G 54.89 1.09 118
>>> 91 hdd 3.64499 1.00000 3726G 2164G 1561G 58.09 1.16 125
>>> 92 hdd 3.64499 1.00000 3726G 1922G 1803G 51.59 1.03 111
>>> 93 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112
>>> 94 hdd 3.64499 1.00000 3726G 1971G 1754G 52.91 1.05 114
>>> 95 hdd 3.64499 1.00000 3726G 1917G 1808G 51.47 1.02 111
>>> 96 hdd 3.64499 1.00000 3726G 1814G 1911G 48.70 0.97 105
>>> 97 hdd 3.64499 1.00000 3726G 1660G 2065G 44.57 0.89 96
>>> 98 hdd 3.64499 1.00000 3726G 1904G 1821G 51.10 1.02 110
>>> 99 hdd 3.64499 1.00000 3726G 2112G 1613G 56.70 1.13 122
>>> 100 hdd 3.64499 1.00000 3726G 1869G 1856G 50.18 1.00 108
>>> 101 hdd 3.64499 1.00000 3726G 2005G 1720G 53.84 1.07 116
>>> 102 hdd 3.64499 1.00000 3726G 2039G 1686G 54.73 1.09 118
>>> 103 hdd 3.64499 1.00000 3726G 2093G 1632G 56.20 1.12 121
>>> 104 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
>>> 105 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
>>> 106 hdd 3.64499 1.00000 3726G 1956G 1769G 52.51 1.05 113
>>> 107 hdd 3.64499 1.00000 3726G 2077G 1648G 55.74 1.11 120
>>> 108 hdd 3.64499 1.00000 3726G 1990G 1735G 53.43 1.06 115
>>> 109 hdd 3.64499 1.00000 3726G 1781G 1944G 47.80 0.95 103
>>> 110 hdd 3.64499 1.00000 3726G 1869G 1856G 50.17 1.00 108
>>> 111 hdd 3.64499 1.00000 3726G 2023G 1702G 54.30 1.08 117
>>> 112 hdd 3.64499 1.00000 3726G 2007G 1718G 53.89 1.07 116
>>> 113 hdd 3.64499 1.00000 3726G 1851G 1874G 49.70 0.99 107
>>> 114 hdd 3.64499 1.00000 3726G 1832G 1893G 49.18 0.98 106
>>> 115 hdd 3.64499 1.00000 3726G 2098G 1627G 56.32 1.12 121
>>> 116 hdd 3.64499 1.00000 3726G 1866G 1859G 50.10 1.00 108
>>> 117 hdd 3.64499 1.00000 3726G 1764G 1961G 47.36 0.94 102
>>> 118 hdd 3.64499 1.00000 3726G 1816G 1909G 48.76 0.97 105
>>> 119 hdd 3.64499 1.00000 3726G 1733G 1992G 46.53 0.93 100
>>> 120 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
>>> 121 hdd 3.64499 1.00000 3726G 1657G 2068G 44.48 0.89 96
>>> 122 hdd 3.64499 1.00000 3726G 1715G 2010G 46.04 0.92 99
>>> 123 hdd 3.64499 1.00000 3726G 1974G 1751G 53.00 1.05 114
>>> 124 hdd 3.64499 1.00000 3726G 1731G 1994G 46.48 0.93 100
>>> 125 hdd 3.64499 1.00000 3726G 1897G 1828G 50.94 1.01 110
>>> 126 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115
>>> 127 hdd 3.64499 1.00000 3726G 1972G 1753G 52.93 1.05 114
>>> 128 hdd 3.64499 1.00000 3726G 1888G 1837G 50.68 1.01 109
>>> 129 hdd 3.64499 1.00000 3726G 2025G 1700G 54.35 1.08 117
>>> 130 hdd 3.64499 1.00000 3726G 1747G 1978G 46.90 0.93 101
>>> 131 hdd 3.64499 1.00000 3726G 2061G 1664G 55.32 1.10 119
>>> 132 hdd 3.64499 1.00000 3726G 2185G 1540G 58.65 1.17 126
>>> 133 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102
>>> 134 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103
>>> 135 hdd 3.64499 1.00000 3726G 1675G 2050G 44.96 0.89 97
>>> 136 hdd 3.64499 1.00000 3726G 1889G 1836G 50.70 1.01 109
>>> 137 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113
>>> 138 hdd 3.64499 1.00000 3726G 1694G 2031G 45.49 0.91 98
>>> 139 hdd 3.64499 1.00000 3726G 2114G 1611G 56.76 1.13 122
>>> 140 hdd 3.64499 1.00000 3726G 1726G 1999G 46.35 0.92 100
>>> 141 hdd 3.64499 1.00000 3726G 1988G 1737G 53.36 1.06 115
>>> 142 hdd 3.64499 1.00000 3726G 2181G 1544G 58.55 1.17 126
>>> 143 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107
>>> 144 hdd 3.64499 1.00000 3726G 2144G 1581G 57.56 1.15 124
>>> 145 hdd 3.64499 1.00000 3726G 1868G 1857G 50.14 1.00 108
>>> 146 hdd 3.64499 1.00000 3726G 2078G 1647G 55.77 1.11 120
>>> 147 hdd 3.64499 1.00000 3726G 1992G 1733G 53.49 1.06 115
>>> 148 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100
>>> 149 hdd 3.64499 1.00000 3726G 2162G 1563G 58.03 1.15 125
>>> 150 hdd 3.64499 1.00000 3726G 2005G 1720G 53.83 1.07 116
>>> 151 hdd 3.64499 1.00000 3726G 1918G 1807G 51.49 1.02 111
>>> 152 hdd 3.64499 1.00000 3726G 1921G 1804G 51.57 1.03 111
>>> 153 hdd 3.64499 1.00000 3726G 1783G 1942G 47.86 0.95 103
>>> 154 hdd 3.64499 1.00000 3726G 2005G 1720G 53.82 1.07 116
>>> 155 hdd 3.64499 1.00000 3726G 1853G 1873G 49.73 0.99 107
>>> 156 hdd 3.64499 1.00000 3726G 1626G 2099G 43.65 0.87 94
>>> 157 hdd 3.64499 1.00000 3726G 1802G 1923G 48.38 0.96 104
>>> 158 hdd 3.64499 1.00000 3726G 1765G 1960G 47.39 0.94 102
>>> 159 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115
>>> 160 hdd 3.64499 1.00000 3726G 1901G 1824G 51.02 1.02 110
>>> 161 hdd 3.64499 1.00000 3726G 1906G 1819G 51.16 1.02 110
>>> 162 hdd 3.64499 1.00000 3726G 1763G 1962G 47.33 0.94 102
>>> 163 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104
>>> 164 hdd 3.64499 1.00000 3726G 1922G 1803G 51.61 1.03 111
>>> 165 hdd 3.64499 1.00000 3726G 1731G 1994G 46.46 0.92 100
>>> 166 hdd 3.64499 1.00000 3726G 2027G 1699G 54.40 1.08 117
>>> 167 hdd 3.64499 1.00000 3726G 1956G 1769G 52.50 1.04 113
>>> 168 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107
>>> 169 hdd 3.64499 1.00000 3726G 1884G 1841G 50.59 1.01 109
>>> 170 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
>>> 171 hdd 3.64499 1.00000 3726G 1898G 1827G 50.96 1.01 110
>>> 172 hdd 3.64499 1.00000 3726G 1781G 1944G 47.82 0.95 103
>>> 173 hdd 3.64499 1.00000 3726G 1628G 2097G 43.71 0.87 94
>>> 174 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
>>> 175 hdd 3.64499 1.00000 3726G 1905G 1820G 51.15 1.02 110
>>> 176 hdd 3.64499 1.00000 3726G 1709G 2016G 45.89 0.91 99
>>> 177 hdd 3.64499 1.00000 3726G 1991G 1734G 53.44 1.06 115
>>> 178 hdd 3.64499 1.00000 3726G 1973G 1752G 52.97 1.05 114
>>> 179 hdd 3.64499 1.00000 3726G 1574G 2151G 42.26 0.84 91
>>> 180 hdd 3.64499 1.00000 3726G 1917G 1808G 51.46 1.02 111
>>> 181 hdd 3.64499 1.00000 3726G 1937G 1788G 52.00 1.03 112
>>> 182 hdd 3.64499 1.00000 3726G 1989G 1736G 53.40 1.06 115
>>> 183 hdd 3.64499 1.00000 3726G 2039G 1686G 54.74 1.09 118
>>> 184 hdd 3.64499 1.00000 3726G 1868G 1857G 50.16 1.00 108
>>> 185 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113
>>> 186 hdd 3.64499 1.00000 3726G 1850G 1875G 49.66 0.99 107
>>> 187 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107
>>> 188 hdd 3.64499 1.00000 3726G 1814G 1911G 48.69 0.97 105
>>> 189 hdd 3.64499 1.00000 3726G 2286G 1439G 61.36 1.22 132
>>> 190 hdd 3.64499 1.00000 3726G 2060G 1665G 55.31 1.10 119
>>> 191 hdd 3.64499 1.00000 3726G 1836G 1889G 49.29 0.98 106
>>> 192 hdd 3.63869 1.00000 3726G 1721G 2004G 46.21 0.92 101
>>> 193 hdd 3.64499 1.00000 3726G 1953G 1772G 52.43 1.04 113
>>> 194 hdd 3.64499 1.00000 3726G 1541G 2184G 41.37 0.82 89
>>> 195 hdd 3.64499 1.00000 3726G 1837G 1888G 49.32 0.98 106
>>> 196 hdd 3.63899 1.00000 3726G 1705G 2020G 45.79 0.91 100
>>> 197 hdd 3.63899 1.00000 3726G 1743G 1982G 46.78 0.93 102
>>> 198 hdd 3.63899 1.00000 3726G 1982G 1743G 53.21 1.06 116
>>> 199 hdd 3.63899 1.00000 3726G 1382G 2343G 37.11 0.74 81
>>> 200 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111
>>> 201 hdd 3.63899 1.00000 3726G 1845G 1880G 49.54 0.99 108
>>> 202 hdd 3.63899 1.00000 3726G 1658G 2067G 44.51 0.89 97
>>> 203 hdd 3.63899 1.00000 3726G 2066G 1659G 55.47 1.10 121
>>> 204 hdd 3.63899 1.00000 3726G 1672G 2053G 44.90 0.89 98
>>> 205 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89
>>> 206 hdd 3.63899 1.00000 3726G 1796G 1929G 48.22 0.96 105
>>> 207 hdd 3.63899 1.00000 3726G 1555G 2170G 41.75 0.83 91
>>> 208 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102
>>> 209 hdd 3.63899 1.00000 3726G 2016G 1709G 54.11 1.08 118
>>> 210 hdd 3.63899 1.00000 3726G 2136G 1589G 57.34 1.14 125
>>> 211 hdd 3.63899 1.00000 3726G 1656G 2069G 44.46 0.88 97
>>> 212 hdd 3.63899 1.00000 3726G 1553G 2172G 41.71 0.83 91
>>> 213 hdd 3.63899 1.00000 3726G 1758G 1967G 47.19 0.94 103
>>> 214 hdd 3.63899 1.00000 3726G 1603G 2122G 43.03 0.86 94
>>> 215 hdd 3.63899 1.00000 3726G 1809G 1916G 48.57 0.97 106
>>> 216 hdd 3.63899 1.00000 3726G 1760G 1965G 47.26 0.94 103
>>> 217 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102
>>> 218 hdd 3.63899 1.00000 3726G 1880G 1845G 50.46 1.00 110
>>> 219 hdd 3.63899 1.00000 3726G 1693G 2032G 45.45 0.90 99
>>> 220 hdd 3.63899 1.00000 3726G 2032G 1693G 54.56 1.09 119
>>> 221 hdd 3.63899 1.00000 3726G 1842G 1883G 49.44 0.98 108
>>> 222 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89
>>> 223 hdd 3.63899 1.00000 3726G 1470G 2255G 39.46 0.79 86
>>> 224 hdd 3.63899 1.00000 3726G 1808G 1917G 48.53 0.97 106
>>> 225 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111
>>> 226 hdd 3.63899 1.00000 3726G 1776G 1949G 47.69 0.95 104
>>> 227 hdd 3.63899 1.00000 3726G 1929G 1796G 51.78 1.03 113
>>> 228 hdd 3.63899 1.00000 3726G 1657G 2068G 44.48 0.89 97
>>> 229 hdd 3.63899 1.00000 3726G 1843G 1882G 49.47 0.98 108
>>> TOTAL 825T 414T 410T 50.24
>>> MIN/MAX VAR: 0.01/1.29 STDDEV: 9.22
>>> --------------------------------------------------------------
>>>
>>> Am 20.10.18 um 20:35 schrieb Janne Johansson:
>>>> Yes, if you have uneven sizes I guess you could end up in a situation
>>>> where you have
>>>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces
>>>> the pool to have one
>>>> PG replica on the 1TB OSD, then it would be possible to state "this
>>>> pool cant write more than X G"
>>>> but when it is full, there would be free space left on some of the
>>>> 2TB-OSDs, but which the pool
>>>> cant utilize. Probably same for uneven OSD hosts if you have those.
>>>>
>>>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth
>>>> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>>>>
>>>>> Dear Janne,
>>>>>
>>>>> yes, of course. But since we only have two pools here, this can not explain the difference.
>>>>> The metadata is replicated (3 copies) across ssd drives, and we have < 3 TB of total raw storage for that.
>>>>> So looking at the raw space usage, we can ignore that.
>>>>>
>>>>> All the rest is used for the ceph_data pool. So the ceph_data pool, in terms of raw storage, is about 50 % used.
>>>>>
>>>>> But in terms of storage shown for that pool, it's almost 63 % %USED.
>>>>> So I guess this can purely be from bad balancing, correct?
>>>>>
>>>>> Cheers,
>>>>> Oliver
>>>>>
>>>>> Am 20.10.18 um 19:49 schrieb Janne Johansson:
>>>>>> Do mind that drives may have more than one pool on them, so RAW space
>>>>>> is what it says, how much free space there is. Then the avail and
>>>>>> %USED on per-pool stats will take replication into account, it can
>>>>>> tell how much data you may write into that particular pool, given that
>>>>>> pools replication or EC settings.
>>>>>>
>>>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth
>>>>>> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>>>>>>
>>>>>>> Dear Cephalopodians,
>>>>>>>
>>>>>>> as many others, I'm also a bit confused by "ceph df" output
>>>>>>> in a pretty straightforward configuration.
>>>>>>>
>>>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile.
>>>>>>>
>>>>>>> I get:
>>>>>>> ----------------------------------------------------------------------------
>>>>>>> # ceph df
>>>>>>> GLOBAL:
>>>>>>> SIZE AVAIL RAW USED %RAW USED
>>>>>>> 824T 410T 414T 50.26
>>>>>>> POOLS:
>>>>>>> NAME ID USED %USED MAX AVAIL OBJECTS
>>>>>>> cephfs_metadata 1 452M 0.05 860G 365774
>>>>>>> cephfs_data 2 275T 62.68 164T 75056403
>>>>>>> ----------------------------------------------------------------------------
>>>>>>>
>>>>>>> So about 50 % of raw space are used, but already ~63 % of filesystem space are used.
>>>>>>> Is this purely from imperfect balancing?
>>>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % usage down to 37.12 %.
>>>>>>>
>>>>>>> We did not yet use the balancer plugin.
>>>>>>> We don't have any pre-luminous clients.
>>>>>>> In that setup, I take it that "upmap" mode would be recommended - correct?
>>>>>>> Any "gotchas" using that on luminous?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Oliver
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
