Am 27.10.18 um 04:12 schrieb Linh Vu: > Should be fine as long as your "mgr/balancer/max_misplaced" is reasonable. I find the default value of 0.05 decent enough, although from experience that seems like 0.05% rather than 5% as suggested here: http://docs.ceph.com/docs/luminous/mgr/balancer/ ; Ok! I did actually choose 0.01. Interestingly, during the initial large rebalancing, it went up to > 2 % of misplaced objects (in small steps) until I decided to stop the balancer for a day to give the cluster enough time to adapt. > You can also choose to turn it on only during certain hours when the cluster might be less busy. The config-keys are there somewhere (there's a post by Dan van der Ster on the ML about them) but they don't actually work in 12.2.8 at least, when I tried them. I suggest just use cron to turn the balancer on and off. I found that mail in the archives. Indeed, that seems helpful. I'll start with permanently leaving the balancer on for now and observe if it has any impact. Since we rarely change the cluster's layout, it should effectively just sit there silently most of the time. Thanks! Oliver > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> > *Sent:* Friday, 26 October 2018 9:32:14 PM > *To:* Linh Vu; Janne Johansson > *Cc:* ceph-users@xxxxxxxxxxxxxx; Peter Wienemann > *Subject:* Re: ceph df space usage confusion - balancing needed? > > Dear Cephalopodians, > > thanks for all your feedback! > > I finally "pushed the button" and let upmap run for ~36 hours. > Previously, we had ~63 % usage of our CephFS with only 50 % raw usage, now, we see only 53.77 % usage. > > That's as close as I expect things to ever become, and we gained about 70 TiB of free storage by that, which is almost one file server. > So the outcome is really close to perfection :-). > > I'm leaving the balancer active now in upmap mode. Any bad experiences with leaving it active "forever"? > > Cheers and many thanks again, > Oliver > > Am 23.10.18 um 01:14 schrieb Linh Vu: >> Upmap is awesome. I ran it on our new cluster before we started ingesting data, so that the PG count is balanced on all OSDs. After ingesting about 315TB, it's still beautifully balanced. Note: we have a few nodes with 8TB OSDs, and the rest on 10TBs. >> >> >> # ceph osd df plain >> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS >> 0 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144 >> 1 mf1hdd 7.27739 1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144 >> 2 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 3 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.37 1.02 144 >> 4 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 5 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141 >> 6 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142 >> 7 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 >> 8 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141 >> 9 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 >> 10 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 >> 11 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.99 1.00 142 >> 12 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141 >> 13 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 14 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141 >> 15 mf1hdd 7.27739 1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144 >> 224 nvmemeta 0.02179 1.00000 22.3GiB 1.52GiB 20.8GiB 6.82 0.24 185 >> 225 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.68 0.24 182 >> 144 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 145 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 146 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 147 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 148 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 149 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 150 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175 >> 151 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 152 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.87 1.00 173 >> 153 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.16 1.01 175 >> 154 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173 >> 155 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 156 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 157 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 158 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174 >> 159 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 242 nvmemeta 0.02179 1.00000 22.3GiB 1.50GiB 20.8GiB 6.70 0.24 182 >> 243 nvmemeta 0.02179 1.00000 22.4GiB 1.45GiB 20.9GiB 6.48 0.23 183 >> 160 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 >> 161 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 162 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 163 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 164 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 165 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 166 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 167 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 168 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 169 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173 >> 170 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175 >> 171 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 172 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 173 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 174 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 175 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 >> 244 nvmemeta 0.02179 1.00000 22.3GiB 1.49GiB 20.9GiB 6.65 0.24 182 >> 245 nvmemeta 0.02179 1.00000 22.4GiB 1.53GiB 20.8GiB 6.86 0.25 185 >> 176 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 177 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 178 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 179 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 180 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 181 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 182 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 183 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.94 1.00 174 >> 184 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 185 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 186 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 187 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 188 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 189 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 190 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 191 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 >> 246 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.64 0.24 184 >> 247 nvmemeta 0.02179 1.00000 22.4GiB 1.47GiB 20.9GiB 6.57 0.24 183 >> 192 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144 >> 193 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.96 1.00 142 >> 194 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.22TiB 28.31 1.01 144 >> 195 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143 >> 196 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 >> 197 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143 >> 198 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.22TiB 28.34 1.01 144 >> 199 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142 >> 200 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143 >> 201 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144 >> 202 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 >> 203 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 >> 204 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144 >> 205 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141 >> 206 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.25TiB 27.79 1.00 141 >> 207 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142 >> 248 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182 >> 249 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.66 0.24 182 >> 208 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 209 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 >> 210 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.74 0.99 141 >> 211 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.13 1.01 143 >> 212 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.93 1.00 142 >> 213 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141 >> 214 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143 >> 215 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141 >> 216 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 >> 217 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 218 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143 >> 219 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 220 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.22TiB 28.22 1.01 143 >> 221 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 >> 222 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143 >> 223 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143 >> 250 nvmemeta 0.02179 1.00000 22.3GiB 1.46GiB 20.9GiB 6.54 0.23 183 >> 251 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.66 0.24 183 >> 16 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144 >> 17 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143 >> 18 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 >> 19 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141 >> 20 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143 >> 21 mf1hdd 7.27739 1.00000 7.28TiB 2.05TiB 5.23TiB 28.17 1.01 143 >> 22 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141 >> 23 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 24 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 >> 25 mf1hdd 7.27739 1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141 >> 26 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142 >> 27 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 >> 28 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 >> 29 mf1hdd 7.27739 1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 >> 30 mf1hdd 7.27739 1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142 >> 31 mf1hdd 7.27739 1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 >> 226 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182 >> 227 nvmemeta 0.02179 1.00000 22.4GiB 1.46GiB 20.9GiB 6.51 0.23 182 >> 32 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 33 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.42TiB 28.00 1.00 174 >> 34 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173 >> 35 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 36 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 37 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 >> 38 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 >> 39 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 40 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 41 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 42 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 43 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173 >> 44 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 >> 45 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 46 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173 >> 47 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 228 nvmemeta 0.02179 1.00000 22.3GiB 1.49GiB 20.9GiB 6.65 0.24 182 >> 229 nvmemeta 0.02179 1.00000 22.4GiB 1.45GiB 20.9GiB 6.47 0.23 183 >> 48 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.17 1.01 175 >> 49 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 50 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 51 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 52 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 53 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 54 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 55 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 56 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 57 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173 >> 58 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 59 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 60 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 61 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 >> 62 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 63 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 >> 230 nvmemeta 0.02179 1.00000 22.3GiB 1.45GiB 20.9GiB 6.48 0.23 183 >> 231 nvmemeta 0.02179 1.00000 22.4GiB 1.48GiB 20.9GiB 6.63 0.24 182 >> 64 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175 >> 65 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 66 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 67 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 68 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 69 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 70 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 >> 71 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 72 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 73 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 74 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 75 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174 >> 76 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 77 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 78 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 79 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 232 nvmemeta 0.02179 1.00000 22.3GiB 1.47GiB 20.9GiB 6.57 0.24 182 >> 233 nvmemeta 0.02179 1.00000 22.4GiB 1.50GiB 20.9GiB 6.70 0.24 184 >> 80 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 81 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 82 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 83 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 84 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173 >> 85 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 86 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 87 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 88 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 89 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 90 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 91 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 92 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 93 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 94 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175 >> 95 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.78 0.99 173 >> 234 nvmemeta 0.02179 1.00000 22.3GiB 1.51GiB 20.8GiB 6.77 0.24 185 >> 235 nvmemeta 0.02179 1.00000 22.4GiB 1.49GiB 20.9GiB 6.65 0.24 182 >> 96 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 97 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 98 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 99 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 >> 100 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 101 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 102 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 103 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.21 1.01 175 >> 104 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 105 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 106 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 107 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 108 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 109 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 110 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 111 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 236 nvmemeta 0.02179 1.00000 22.3GiB 1.46GiB 20.9GiB 6.53 0.23 185 >> 237 nvmemeta 0.02179 1.00000 22.4GiB 1.46GiB 20.9GiB 6.54 0.23 182 >> 112 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 113 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 114 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 >> 115 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 116 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 117 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 118 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 119 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.96 1.00 174 >> 120 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 121 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 122 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 >> 123 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174 >> 124 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 125 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 >> 126 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 >> 127 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 238 nvmemeta 0.02179 1.00000 22.3GiB 1.48GiB 20.9GiB 6.62 0.24 182 >> 239 nvmemeta 0.02179 1.00000 22.4GiB 1.51GiB 20.8GiB 6.75 0.24 182 >> 128 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 >> 129 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 130 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 >> 131 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 132 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 133 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 134 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 135 mf1hdd 8.91019 1.00000 8.91TiB 2.51TiB 6.40TiB 28.13 1.01 175 >> 136 mf1hdd 8.91019 1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 >> 137 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 138 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 139 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 >> 140 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 >> 141 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 142 mf1hdd 8.91019 1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174 >> 143 mf1hdd 8.91019 1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 >> 240 nvmemeta 0.02179 1.00000 22.3GiB 1.61GiB 20.7GiB 7.22 0.26 184 >> 241 nvmemeta 0.02179 1.00000 22.4GiB 1.43GiB 20.9GiB 6.41 0.23 182 >> TOTAL 1.85PiB 528TiB 1.33PiB 27.93 >> MIN/MAX VAR: 0.23/1.02 STDDEV: 7.10 >> >> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ >> *From:* ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> >> *Sent:* Sunday, 21 October 2018 6:57:49 AM >> *To:* Janne Johansson >> *Cc:* ceph-users@xxxxxxxxxxxxxx; Peter Wienemann >> *Subject:* Re: ceph df space usage confusion - balancing needed? >> >> Ok, I'll try out the balancer end of the upcoming week then (after we've fixed a HW-issue with one of our mons >> and the cooling system). >> >> Until then, any further advice and whether upmap is recommended over crush-compat (all clients are Luminous) are welcome ;-). >> >> Cheers, >> Oliver >> >> Am 20.10.18 um 21:26 schrieb Janne Johansson: >>> Ok, can't say "why" then, I'd reweigh them somewhat to even it out, >>> 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for >>> the MGRs, a script or just a few manual tweaks might be in order. >>> >>> Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth >>> <freyermuth@xxxxxxxxxxxxxxxxxx>: >>>> >>>> All OSDs are of the very same size. One OSD host has slightly more disks (33 instead of 31), though. >>>> So also that that can't explain the hefty difference. >>>> >>>> I attach the output of "ceph osd tree" and "ceph osd df". >>>> >>>> The crush rule for the ceph_data pool is: >>>> rule cephfs_data { >>>> id 2 >>>> type erasure >>>> min_size 3 >>>> max_size 6 >>>> step set_chooseleaf_tries 5 >>>> step set_choose_tries 100 >>>> step take default class hdd >>>> step chooseleaf indep 0 type host >>>> step emit >>>> } >>>> So that only considers the hdd device class. EC is done with k=4 m=2. >>>> >>>> So I don't see any imbalance on the hardware level, but only a somewhat uneven distribution of PGs. >>>> Am I missing something, or is this really just a case for the ceph balancer plugin? >>>> I'm just a bit astonished this effect is so huge. >>>> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even distribution without balancing? >>>> But it yields about 100 PGs per OSD, as you can see... >>>> >>>> -------------------------------------------------------------- >>>> # ceph osd tree >>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >>>> -1 826.26428 root default >>>> -3 0.43700 host mon001 >>>> 0 ssd 0.21799 osd.0 up 1.00000 1.00000 >>>> 1 ssd 0.21799 osd.1 up 1.00000 1.00000 >>>> -5 0.43700 host mon002 >>>> 2 ssd 0.21799 osd.2 up 1.00000 1.00000 >>>> 3 ssd 0.21799 osd.3 up 1.00000 1.00000 >>>> -31 1.81898 host mon003 >>>> 230 ssd 0.90999 osd.230 up 1.00000 1.00000 >>>> 231 ssd 0.90999 osd.231 up 1.00000 1.00000 >>>> -10 116.64600 host osd001 >>>> 4 hdd 3.64499 osd.4 up 1.00000 1.00000 >>>> 5 hdd 3.64499 osd.5 up 1.00000 1.00000 >>>> 6 hdd 3.64499 osd.6 up 1.00000 1.00000 >>>> 7 hdd 3.64499 osd.7 up 1.00000 1.00000 >>>> 8 hdd 3.64499 osd.8 up 1.00000 1.00000 >>>> 9 hdd 3.64499 osd.9 up 1.00000 1.00000 >>>> 10 hdd 3.64499 osd.10 up 1.00000 1.00000 >>>> 11 hdd 3.64499 osd.11 up 1.00000 1.00000 >>>> 12 hdd 3.64499 osd.12 up 1.00000 1.00000 >>>> 13 hdd 3.64499 osd.13 up 1.00000 1.00000 >>>> 14 hdd 3.64499 osd.14 up 1.00000 1.00000 >>>> 15 hdd 3.64499 osd.15 up 1.00000 1.00000 >>>> 16 hdd 3.64499 osd.16 up 1.00000 1.00000 >>>> 17 hdd 3.64499 osd.17 up 1.00000 1.00000 >>>> 18 hdd 3.64499 osd.18 up 1.00000 1.00000 >>>> 19 hdd 3.64499 osd.19 up 1.00000 1.00000 >>>> 20 hdd 3.64499 osd.20 up 1.00000 1.00000 >>>> 21 hdd 3.64499 osd.21 up 1.00000 1.00000 >>>> 22 hdd 3.64499 osd.22 up 1.00000 1.00000 >>>> 23 hdd 3.64499 osd.23 up 1.00000 1.00000 >>>> 24 hdd 3.64499 osd.24 up 1.00000 1.00000 >>>> 25 hdd 3.64499 osd.25 up 1.00000 1.00000 >>>> 26 hdd 3.64499 osd.26 up 1.00000 1.00000 >>>> 27 hdd 3.64499 osd.27 up 1.00000 1.00000 >>>> 28 hdd 3.64499 osd.28 up 1.00000 1.00000 >>>> 29 hdd 3.64499 osd.29 up 1.00000 1.00000 >>>> 30 hdd 3.64499 osd.30 up 1.00000 1.00000 >>>> 31 hdd 3.64499 osd.31 up 1.00000 1.00000 >>>> 32 hdd 3.64499 osd.32 up 1.00000 1.00000 >>>> 33 hdd 3.64499 osd.33 up 1.00000 1.00000 >>>> 34 hdd 3.64499 osd.34 up 1.00000 1.00000 >>>> 35 hdd 3.64499 osd.35 up 1.00000 1.00000 >>>> -13 116.64600 host osd002 >>>> 36 hdd 3.64499 osd.36 up 1.00000 1.00000 >>>> 37 hdd 3.64499 osd.37 up 1.00000 1.00000 >>>> 38 hdd 3.64499 osd.38 up 1.00000 1.00000 >>>> 39 hdd 3.64499 osd.39 up 1.00000 1.00000 >>>> 40 hdd 3.64499 osd.40 up 1.00000 1.00000 >>>> 41 hdd 3.64499 osd.41 up 1.00000 1.00000 >>>> 42 hdd 3.64499 osd.42 up 1.00000 1.00000 >>>> 43 hdd 3.64499 osd.43 up 1.00000 1.00000 >>>> 44 hdd 3.64499 osd.44 up 1.00000 1.00000 >>>> 45 hdd 3.64499 osd.45 up 1.00000 1.00000 >>>> 46 hdd 3.64499 osd.46 up 1.00000 1.00000 >>>> 47 hdd 3.64499 osd.47 up 1.00000 1.00000 >>>> 48 hdd 3.64499 osd.48 up 1.00000 1.00000 >>>> 49 hdd 3.64499 osd.49 up 1.00000 1.00000 >>>> 50 hdd 3.64499 osd.50 up 1.00000 1.00000 >>>> 51 hdd 3.64499 osd.51 up 1.00000 1.00000 >>>> 52 hdd 3.64499 osd.52 up 1.00000 1.00000 >>>> 53 hdd 3.64499 osd.53 up 1.00000 1.00000 >>>> 54 hdd 3.64499 osd.54 up 1.00000 1.00000 >>>> 55 hdd 3.64499 osd.55 up 1.00000 1.00000 >>>> 56 hdd 3.64499 osd.56 up 1.00000 1.00000 >>>> 57 hdd 3.64499 osd.57 up 1.00000 1.00000 >>>> 58 hdd 3.64499 osd.58 up 1.00000 1.00000 >>>> 59 hdd 3.64499 osd.59 up 1.00000 1.00000 >>>> 60 hdd 3.64499 osd.60 up 1.00000 1.00000 >>>> 61 hdd 3.64499 osd.61 up 1.00000 1.00000 >>>> 62 hdd 3.64499 osd.62 up 1.00000 1.00000 >>>> 63 hdd 3.64499 osd.63 up 1.00000 1.00000 >>>> 64 hdd 3.64499 osd.64 up 1.00000 1.00000 >>>> 65 hdd 3.64499 osd.65 up 1.00000 1.00000 >>>> 66 hdd 3.64499 osd.66 up 1.00000 1.00000 >>>> 67 hdd 3.64499 osd.67 up 1.00000 1.00000 >>>> -16 116.63899 host osd003 >>>> 68 hdd 3.64499 osd.68 up 1.00000 1.00000 >>>> 69 hdd 3.64499 osd.69 up 1.00000 1.00000 >>>> 70 hdd 3.64499 osd.70 up 1.00000 1.00000 >>>> 71 hdd 3.64499 osd.71 up 1.00000 1.00000 >>>> 72 hdd 3.64499 osd.72 up 1.00000 1.00000 >>>> 73 hdd 3.64499 osd.73 up 1.00000 1.00000 >>>> 74 hdd 3.64499 osd.74 up 1.00000 1.00000 >>>> 75 hdd 3.64499 osd.75 up 1.00000 1.00000 >>>> 76 hdd 3.64499 osd.76 up 1.00000 1.00000 >>>> 77 hdd 3.64499 osd.77 up 1.00000 1.00000 >>>> 78 hdd 3.64499 osd.78 up 1.00000 1.00000 >>>> 79 hdd 3.64499 osd.79 up 1.00000 1.00000 >>>> 80 hdd 3.63899 osd.80 up 1.00000 1.00000 >>>> 81 hdd 3.64499 osd.81 up 1.00000 1.00000 >>>> 82 hdd 3.64499 osd.82 up 1.00000 1.00000 >>>> 83 hdd 3.64499 osd.83 up 1.00000 1.00000 >>>> 84 hdd 3.64499 osd.84 up 1.00000 1.00000 >>>> 85 hdd 3.64499 osd.85 up 1.00000 1.00000 >>>> 86 hdd 3.64499 osd.86 up 1.00000 1.00000 >>>> 87 hdd 3.64499 osd.87 up 1.00000 1.00000 >>>> 88 hdd 3.64499 osd.88 up 1.00000 1.00000 >>>> 89 hdd 3.64499 osd.89 up 1.00000 1.00000 >>>> 90 hdd 3.64499 osd.90 up 1.00000 1.00000 >>>> 91 hdd 3.64499 osd.91 up 1.00000 1.00000 >>>> 92 hdd 3.64499 osd.92 up 1.00000 1.00000 >>>> 93 hdd 3.64499 osd.93 up 1.00000 1.00000 >>>> 94 hdd 3.64499 osd.94 up 1.00000 1.00000 >>>> 95 hdd 3.64499 osd.95 up 1.00000 1.00000 >>>> 96 hdd 3.64499 osd.96 up 1.00000 1.00000 >>>> 97 hdd 3.64499 osd.97 up 1.00000 1.00000 >>>> 98 hdd 3.64499 osd.98 up 1.00000 1.00000 >>>> 99 hdd 3.64499 osd.99 up 1.00000 1.00000 >>>> -19 116.64600 host osd004 >>>> 100 hdd 3.64499 osd.100 up 1.00000 1.00000 >>>> 101 hdd 3.64499 osd.101 up 1.00000 1.00000 >>>> 102 hdd 3.64499 osd.102 up 1.00000 1.00000 >>>> 103 hdd 3.64499 osd.103 up 1.00000 1.00000 >>>> 104 hdd 3.64499 osd.104 up 1.00000 1.00000 >>>> 105 hdd 3.64499 osd.105 up 1.00000 1.00000 >>>> 106 hdd 3.64499 osd.106 up 1.00000 1.00000 >>>> 107 hdd 3.64499 osd.107 up 1.00000 1.00000 >>>> 108 hdd 3.64499 osd.108 up 1.00000 1.00000 >>>> 109 hdd 3.64499 osd.109 up 1.00000 1.00000 >>>> 110 hdd 3.64499 osd.110 up 1.00000 1.00000 >>>> 111 hdd 3.64499 osd.111 up 1.00000 1.00000 >>>> 112 hdd 3.64499 osd.112 up 1.00000 1.00000 >>>> 113 hdd 3.64499 osd.113 up 1.00000 1.00000 >>>> 114 hdd 3.64499 osd.114 up 1.00000 1.00000 >>>> 115 hdd 3.64499 osd.115 up 1.00000 1.00000 >>>> 116 hdd 3.64499 osd.116 up 1.00000 1.00000 >>>> 117 hdd 3.64499 osd.117 up 1.00000 1.00000 >>>> 118 hdd 3.64499 osd.118 up 1.00000 1.00000 >>>> 119 hdd 3.64499 osd.119 up 1.00000 1.00000 >>>> 120 hdd 3.64499 osd.120 up 1.00000 1.00000 >>>> 121 hdd 3.64499 osd.121 up 1.00000 1.00000 >>>> 122 hdd 3.64499 osd.122 up 1.00000 1.00000 >>>> 123 hdd 3.64499 osd.123 up 1.00000 1.00000 >>>> 124 hdd 3.64499 osd.124 up 1.00000 1.00000 >>>> 125 hdd 3.64499 osd.125 up 1.00000 1.00000 >>>> 126 hdd 3.64499 osd.126 up 1.00000 1.00000 >>>> 127 hdd 3.64499 osd.127 up 1.00000 1.00000 >>>> 128 hdd 3.64499 osd.128 up 1.00000 1.00000 >>>> 129 hdd 3.64499 osd.129 up 1.00000 1.00000 >>>> 130 hdd 3.64499 osd.130 up 1.00000 1.00000 >>>> 131 hdd 3.64499 osd.131 up 1.00000 1.00000 >>>> -22 116.64600 host osd005 >>>> 132 hdd 3.64499 osd.132 up 1.00000 1.00000 >>>> 133 hdd 3.64499 osd.133 up 1.00000 1.00000 >>>> 134 hdd 3.64499 osd.134 up 1.00000 1.00000 >>>> 135 hdd 3.64499 osd.135 up 1.00000 1.00000 >>>> 136 hdd 3.64499 osd.136 up 1.00000 1.00000 >>>> 137 hdd 3.64499 osd.137 up 1.00000 1.00000 >>>> 138 hdd 3.64499 osd.138 up 1.00000 1.00000 >>>> 139 hdd 3.64499 osd.139 up 1.00000 1.00000 >>>> 140 hdd 3.64499 osd.140 up 1.00000 1.00000 >>>> 141 hdd 3.64499 osd.141 up 1.00000 1.00000 >>>> 142 hdd 3.64499 osd.142 up 1.00000 1.00000 >>>> 143 hdd 3.64499 osd.143 up 1.00000 1.00000 >>>> 144 hdd 3.64499 osd.144 up 1.00000 1.00000 >>>> 145 hdd 3.64499 osd.145 up 1.00000 1.00000 >>>> 146 hdd 3.64499 osd.146 up 1.00000 1.00000 >>>> 147 hdd 3.64499 osd.147 up 1.00000 1.00000 >>>> 148 hdd 3.64499 osd.148 up 1.00000 1.00000 >>>> 149 hdd 3.64499 osd.149 up 1.00000 1.00000 >>>> 150 hdd 3.64499 osd.150 up 1.00000 1.00000 >>>> 151 hdd 3.64499 osd.151 up 1.00000 1.00000 >>>> 152 hdd 3.64499 osd.152 up 1.00000 1.00000 >>>> 153 hdd 3.64499 osd.153 up 1.00000 1.00000 >>>> 154 hdd 3.64499 osd.154 up 1.00000 1.00000 >>>> 155 hdd 3.64499 osd.155 up 1.00000 1.00000 >>>> 156 hdd 3.64499 osd.156 up 1.00000 1.00000 >>>> 157 hdd 3.64499 osd.157 up 1.00000 1.00000 >>>> 158 hdd 3.64499 osd.158 up 1.00000 1.00000 >>>> 159 hdd 3.64499 osd.159 up 1.00000 1.00000 >>>> 160 hdd 3.64499 osd.160 up 1.00000 1.00000 >>>> 161 hdd 3.64499 osd.161 up 1.00000 1.00000 >>>> 162 hdd 3.64499 osd.162 up 1.00000 1.00000 >>>> 163 hdd 3.64499 osd.163 up 1.00000 1.00000 >>>> -25 116.63335 host osd006 >>>> 164 hdd 3.64499 osd.164 up 1.00000 1.00000 >>>> 165 hdd 3.64499 osd.165 up 1.00000 1.00000 >>>> 166 hdd 3.64499 osd.166 up 1.00000 1.00000 >>>> 167 hdd 3.64499 osd.167 up 1.00000 1.00000 >>>> 168 hdd 3.64499 osd.168 up 1.00000 1.00000 >>>> 169 hdd 3.64499 osd.169 up 1.00000 1.00000 >>>> 170 hdd 3.64499 osd.170 up 1.00000 1.00000 >>>> 171 hdd 3.64499 osd.171 up 1.00000 1.00000 >>>> 172 hdd 3.64499 osd.172 up 1.00000 1.00000 >>>> 173 hdd 3.64499 osd.173 up 1.00000 1.00000 >>>> 174 hdd 3.64499 osd.174 up 1.00000 1.00000 >>>> 175 hdd 3.64499 osd.175 up 1.00000 1.00000 >>>> 176 hdd 3.64499 osd.176 up 1.00000 1.00000 >>>> 177 hdd 3.64499 osd.177 up 1.00000 1.00000 >>>> 178 hdd 3.64499 osd.178 up 1.00000 1.00000 >>>> 179 hdd 3.64499 osd.179 up 1.00000 1.00000 >>>> 180 hdd 3.64499 osd.180 up 1.00000 1.00000 >>>> 181 hdd 3.64499 osd.181 up 1.00000 1.00000 >>>> 182 hdd 3.64499 osd.182 up 1.00000 1.00000 >>>> 183 hdd 3.64499 osd.183 up 1.00000 1.00000 >>>> 184 hdd 3.64499 osd.184 up 1.00000 1.00000 >>>> 185 hdd 3.64499 osd.185 up 1.00000 1.00000 >>>> 186 hdd 3.64499 osd.186 up 1.00000 1.00000 >>>> 187 hdd 3.64499 osd.187 up 1.00000 1.00000 >>>> 188 hdd 3.64499 osd.188 up 1.00000 1.00000 >>>> 189 hdd 3.64499 osd.189 up 1.00000 1.00000 >>>> 190 hdd 3.64499 osd.190 up 1.00000 1.00000 >>>> 191 hdd 3.64499 osd.191 up 1.00000 1.00000 >>>> 192 hdd 3.63869 osd.192 up 1.00000 1.00000 >>>> 193 hdd 3.64499 osd.193 up 1.00000 1.00000 >>>> 194 hdd 3.64499 osd.194 up 1.00000 1.00000 >>>> 195 hdd 3.64499 osd.195 up 1.00000 1.00000 >>>> -28 123.71500 host osd007 >>>> 196 hdd 3.63899 osd.196 up 1.00000 1.00000 >>>> 197 hdd 3.63899 osd.197 up 1.00000 1.00000 >>>> 198 hdd 3.63899 osd.198 up 1.00000 1.00000 >>>> 199 hdd 3.63899 osd.199 up 1.00000 1.00000 >>>> 200 hdd 3.63899 osd.200 up 1.00000 1.00000 >>>> 201 hdd 3.63899 osd.201 up 1.00000 1.00000 >>>> 202 hdd 3.63899 osd.202 up 1.00000 1.00000 >>>> 203 hdd 3.63899 osd.203 up 1.00000 1.00000 >>>> 204 hdd 3.63899 osd.204 up 1.00000 1.00000 >>>> 205 hdd 3.63899 osd.205 up 1.00000 1.00000 >>>> 206 hdd 3.63899 osd.206 up 1.00000 1.00000 >>>> 207 hdd 3.63899 osd.207 up 1.00000 1.00000 >>>> 208 hdd 3.63899 osd.208 up 1.00000 1.00000 >>>> 209 hdd 3.63899 osd.209 up 1.00000 1.00000 >>>> 210 hdd 3.63899 osd.210 up 1.00000 1.00000 >>>> 211 hdd 3.63899 osd.211 up 1.00000 1.00000 >>>> 212 hdd 3.63899 osd.212 up 1.00000 1.00000 >>>> 213 hdd 3.63899 osd.213 up 1.00000 1.00000 >>>> 214 hdd 3.63899 osd.214 up 1.00000 1.00000 >>>> 215 hdd 3.63899 osd.215 up 1.00000 1.00000 >>>> 216 hdd 3.63899 osd.216 up 1.00000 1.00000 >>>> 217 hdd 3.63899 osd.217 up 1.00000 1.00000 >>>> 218 hdd 3.63899 osd.218 up 1.00000 1.00000 >>>> 219 hdd 3.63899 osd.219 up 1.00000 1.00000 >>>> 220 hdd 3.63899 osd.220 up 1.00000 1.00000 >>>> 221 hdd 3.63899 osd.221 up 1.00000 1.00000 >>>> 222 hdd 3.63899 osd.222 up 1.00000 1.00000 >>>> 223 hdd 3.63899 osd.223 up 1.00000 1.00000 >>>> 224 hdd 3.63899 osd.224 up 1.00000 1.00000 >>>> 225 hdd 3.63899 osd.225 up 1.00000 1.00000 >>>> 226 hdd 3.63899 osd.226 up 1.00000 1.00000 >>>> 227 hdd 3.63899 osd.227 up 1.00000 1.00000 >>>> 228 hdd 3.63899 osd.228 up 1.00000 1.00000 >>>> 229 hdd 3.63899 osd.229 up 1.00000 1.00000 >>>> >>>> # ceph osd df >>>> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS >>>> 0 ssd 0.21799 1.00000 223G 4305M 219G 1.88 0.04 64 >>>> 1 ssd 0.21799 1.00000 223G 3353M 220G 1.46 0.03 64 >>>> 2 ssd 0.21799 1.00000 223G 1836M 221G 0.80 0.02 56 >>>> 3 ssd 0.21799 1.00000 223G 2276M 221G 0.99 0.02 72 >>>> 230 ssd 0.90999 1.00000 931G 2775M 928G 0.29 0.01 60 >>>> 231 ssd 0.90999 1.00000 931G 3279M 928G 0.34 0.01 68 >>>> 4 hdd 3.64499 1.00000 3726G 1916G 1809G 51.45 1.02 111 >>>> 5 hdd 3.64499 1.00000 3726G 2144G 1581G 57.54 1.15 124 >>>> 6 hdd 3.64499 1.00000 3726G 1887G 1838G 50.66 1.01 109 >>>> 7 hdd 3.64499 1.00000 3726G 1642G 2083G 44.09 0.88 95 >>>> 8 hdd 3.64499 1.00000 3726G 1969G 1756G 52.86 1.05 114 >>>> 9 hdd 3.64499 1.00000 3726G 1577G 2148G 42.34 0.84 91 >>>> 10 hdd 3.64499 1.00000 3726G 2043G 1682G 54.85 1.09 118 >>>> 11 hdd 3.64499 1.00000 3726G 1957G 1768G 52.54 1.05 113 >>>> 12 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114 >>>> 13 hdd 3.64499 1.00000 3726G 1676G 2049G 45.00 0.90 97 >>>> 14 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102 >>>> 15 hdd 3.64499 1.00000 3726G 2161G 1564G 58.02 1.15 125 >>>> 16 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101 >>>> 17 hdd 3.64499 1.00000 3726G 2022G 1703G 54.29 1.08 117 >>>> 18 hdd 3.64499 1.00000 3726G 1784G 1941G 47.89 0.95 103 >>>> 19 hdd 3.64499 1.00000 3726G 1783G 1942G 47.85 0.95 103 >>>> 20 hdd 3.64499 1.00000 3726G 1731G 1994G 46.47 0.92 100 >>>> 21 hdd 3.64499 1.00000 3726G 1939G 1786G 52.06 1.04 112 >>>> 22 hdd 3.64499 1.00000 3726G 2061G 1664G 55.34 1.10 119 >>>> 23 hdd 3.64499 1.00000 3726G 1538G 2187G 41.29 0.82 89 >>>> 24 hdd 3.64499 1.00000 3726G 2168G 1557G 58.19 1.16 125 >>>> 25 hdd 3.64499 1.00000 3726G 2023G 1702G 54.32 1.08 117 >>>> 26 hdd 3.64499 1.00000 3726G 1900G 1825G 51.00 1.02 110 >>>> 27 hdd 3.64499 1.00000 3726G 2126G 1599G 57.08 1.14 123 >>>> 28 hdd 3.64499 1.00000 3726G 1938G 1787G 52.02 1.04 112 >>>> 29 hdd 3.64499 1.00000 3726G 1663G 2062G 44.65 0.89 96 >>>> 30 hdd 3.64499 1.00000 3726G 2163G 1562G 58.07 1.16 125 >>>> 31 hdd 3.64499 1.00000 3726G 1919G 1806G 51.52 1.03 111 >>>> 32 hdd 3.64499 1.00000 3726G 1696G 2029G 45.52 0.91 98 >>>> 33 hdd 3.64499 1.00000 3726G 1438G 2287G 38.61 0.77 83 >>>> 34 hdd 3.64499 1.00000 3726G 2073G 1652G 55.64 1.11 120 >>>> 35 hdd 3.64499 1.00000 3726G 2079G 1646G 55.81 1.11 120 >>>> 36 hdd 3.64499 1.00000 3726G 1955G 1770G 52.50 1.04 113 >>>> 37 hdd 3.64499 1.00000 3726G 2180G 1545G 58.51 1.16 126 >>>> 38 hdd 3.64499 1.00000 3726G 1664G 2061G 44.66 0.89 96 >>>> 39 hdd 3.64499 1.00000 3726G 2003G 1722G 53.77 1.07 116 >>>> 40 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103 >>>> 41 hdd 3.64499 1.00000 3726G 1903G 1822G 51.08 1.02 110 >>>> 42 hdd 3.64499 1.00000 3726G 1747G 1978G 46.89 0.93 101 >>>> 43 hdd 3.64499 1.00000 3726G 2021G 1704G 54.26 1.08 117 >>>> 44 hdd 3.64499 1.00000 3726G 1852G 1873G 49.71 0.99 107 >>>> 45 hdd 3.64499 1.00000 3726G 1522G 2203G 40.87 0.81 88 >>>> 46 hdd 3.64499 1.00000 3726G 1851G 1874G 49.69 0.99 107 >>>> 47 hdd 3.64499 1.00000 3726G 1698G 2027G 45.59 0.91 98 >>>> 48 hdd 3.64499 1.00000 3726G 1940G 1785G 52.08 1.04 112 >>>> 49 hdd 3.64499 1.00000 3726G 1746G 1979G 46.88 0.93 101 >>>> 50 hdd 3.64499 1.00000 3726G 1728G 1997G 46.40 0.92 100 >>>> 51 hdd 3.64499 1.00000 3726G 2093G 1632G 56.18 1.12 121 >>>> 52 hdd 3.64499 1.00000 3726G 2235G 1490G 60.01 1.19 129 >>>> 53 hdd 3.64499 1.00000 3726G 1871G 1854G 50.23 1.00 108 >>>> 54 hdd 3.64499 1.00000 3726G 2094G 1631G 56.22 1.12 121 >>>> 55 hdd 3.64499 1.00000 3726G 2422G 1303G 65.02 1.29 140 >>>> 56 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 >>>> 57 hdd 3.64499 1.00000 3726G 1729G 1996G 46.43 0.92 100 >>>> 58 hdd 3.64499 1.00000 3726G 1976G 1749G 53.05 1.06 114 >>>> 59 hdd 3.64499 1.00000 3726G 1954G 1771G 52.46 1.04 113 >>>> 60 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107 >>>> 61 hdd 3.64499 1.00000 3726G 2042G 1683G 54.82 1.09 118 >>>> 62 hdd 3.64499 1.00000 3726G 1921G 1804G 51.58 1.03 111 >>>> 63 hdd 3.64499 1.00000 3726G 2128G 1597G 57.13 1.14 123 >>>> 64 hdd 3.64499 1.00000 3726G 2037G 1688G 54.67 1.09 118 >>>> 65 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114 >>>> 66 hdd 3.64499 1.00000 3726G 1539G 2186G 41.32 0.82 89 >>>> 67 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 >>>> 68 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 >>>> 69 hdd 3.64499 1.00000 3726G 2092G 1633G 56.16 1.12 121 >>>> 70 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104 >>>> 71 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129 >>>> 72 hdd 3.64499 1.00000 3726G 1557G 2168G 41.79 0.83 90 >>>> 73 hdd 3.64499 1.00000 3726G 2300G 1425G 61.75 1.23 133 >>>> 74 hdd 3.64499 1.00000 3726G 2132G 1593G 57.24 1.14 123 >>>> 75 hdd 3.64499 1.00000 3726G 1818G 1907G 48.80 0.97 105 >>>> 76 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129 >>>> 77 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101 >>>> 78 hdd 3.64499 1.00000 3726G 1766G 1959G 47.41 0.94 102 >>>> 79 hdd 3.64499 1.00000 3726G 1678G 2047G 45.04 0.90 97 >>>> 80 hdd 3.63899 1.00000 3726G 1744G 1981G 46.82 0.93 102 >>>> 81 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 >>>> 82 hdd 3.64499 1.00000 3726G 1711G 2014G 45.93 0.91 99 >>>> 83 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100 >>>> 84 hdd 3.64499 1.00000 3726G 1834G 1891G 49.22 0.98 106 >>>> 85 hdd 3.64499 1.00000 3726G 1709G 2016G 45.88 0.91 99 >>>> 86 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112 >>>> 87 hdd 3.64499 1.00000 3726G 2131G 1594G 57.19 1.14 123 >>>> 88 hdd 3.64499 1.00000 3726G 1559G 2166G 41.85 0.83 90 >>>> 89 hdd 3.64499 1.00000 3726G 1645G 2080G 44.16 0.88 95 >>>> 90 hdd 3.64499 1.00000 3726G 2045G 1680G 54.89 1.09 118 >>>> 91 hdd 3.64499 1.00000 3726G 2164G 1561G 58.09 1.16 125 >>>> 92 hdd 3.64499 1.00000 3726G 1922G 1803G 51.59 1.03 111 >>>> 93 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112 >>>> 94 hdd 3.64499 1.00000 3726G 1971G 1754G 52.91 1.05 114 >>>> 95 hdd 3.64499 1.00000 3726G 1917G 1808G 51.47 1.02 111 >>>> 96 hdd 3.64499 1.00000 3726G 1814G 1911G 48.70 0.97 105 >>>> 97 hdd 3.64499 1.00000 3726G 1660G 2065G 44.57 0.89 96 >>>> 98 hdd 3.64499 1.00000 3726G 1904G 1821G 51.10 1.02 110 >>>> 99 hdd 3.64499 1.00000 3726G 2112G 1613G 56.70 1.13 122 >>>> 100 hdd 3.64499 1.00000 3726G 1869G 1856G 50.18 1.00 108 >>>> 101 hdd 3.64499 1.00000 3726G 2005G 1720G 53.84 1.07 116 >>>> 102 hdd 3.64499 1.00000 3726G 2039G 1686G 54.73 1.09 118 >>>> 103 hdd 3.64499 1.00000 3726G 2093G 1632G 56.20 1.12 121 >>>> 104 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 >>>> 105 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 >>>> 106 hdd 3.64499 1.00000 3726G 1956G 1769G 52.51 1.05 113 >>>> 107 hdd 3.64499 1.00000 3726G 2077G 1648G 55.74 1.11 120 >>>> 108 hdd 3.64499 1.00000 3726G 1990G 1735G 53.43 1.06 115 >>>> 109 hdd 3.64499 1.00000 3726G 1781G 1944G 47.80 0.95 103 >>>> 110 hdd 3.64499 1.00000 3726G 1869G 1856G 50.17 1.00 108 >>>> 111 hdd 3.64499 1.00000 3726G 2023G 1702G 54.30 1.08 117 >>>> 112 hdd 3.64499 1.00000 3726G 2007G 1718G 53.89 1.07 116 >>>> 113 hdd 3.64499 1.00000 3726G 1851G 1874G 49.70 0.99 107 >>>> 114 hdd 3.64499 1.00000 3726G 1832G 1893G 49.18 0.98 106 >>>> 115 hdd 3.64499 1.00000 3726G 2098G 1627G 56.32 1.12 121 >>>> 116 hdd 3.64499 1.00000 3726G 1866G 1859G 50.10 1.00 108 >>>> 117 hdd 3.64499 1.00000 3726G 1764G 1961G 47.36 0.94 102 >>>> 118 hdd 3.64499 1.00000 3726G 1816G 1909G 48.76 0.97 105 >>>> 119 hdd 3.64499 1.00000 3726G 1733G 1992G 46.53 0.93 100 >>>> 120 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 >>>> 121 hdd 3.64499 1.00000 3726G 1657G 2068G 44.48 0.89 96 >>>> 122 hdd 3.64499 1.00000 3726G 1715G 2010G 46.04 0.92 99 >>>> 123 hdd 3.64499 1.00000 3726G 1974G 1751G 53.00 1.05 114 >>>> 124 hdd 3.64499 1.00000 3726G 1731G 1994G 46.48 0.93 100 >>>> 125 hdd 3.64499 1.00000 3726G 1897G 1828G 50.94 1.01 110 >>>> 126 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115 >>>> 127 hdd 3.64499 1.00000 3726G 1972G 1753G 52.93 1.05 114 >>>> 128 hdd 3.64499 1.00000 3726G 1888G 1837G 50.68 1.01 109 >>>> 129 hdd 3.64499 1.00000 3726G 2025G 1700G 54.35 1.08 117 >>>> 130 hdd 3.64499 1.00000 3726G 1747G 1978G 46.90 0.93 101 >>>> 131 hdd 3.64499 1.00000 3726G 2061G 1664G 55.32 1.10 119 >>>> 132 hdd 3.64499 1.00000 3726G 2185G 1540G 58.65 1.17 126 >>>> 133 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102 >>>> 134 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103 >>>> 135 hdd 3.64499 1.00000 3726G 1675G 2050G 44.96 0.89 97 >>>> 136 hdd 3.64499 1.00000 3726G 1889G 1836G 50.70 1.01 109 >>>> 137 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113 >>>> 138 hdd 3.64499 1.00000 3726G 1694G 2031G 45.49 0.91 98 >>>> 139 hdd 3.64499 1.00000 3726G 2114G 1611G 56.76 1.13 122 >>>> 140 hdd 3.64499 1.00000 3726G 1726G 1999G 46.35 0.92 100 >>>> 141 hdd 3.64499 1.00000 3726G 1988G 1737G 53.36 1.06 115 >>>> 142 hdd 3.64499 1.00000 3726G 2181G 1544G 58.55 1.17 126 >>>> 143 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107 >>>> 144 hdd 3.64499 1.00000 3726G 2144G 1581G 57.56 1.15 124 >>>> 145 hdd 3.64499 1.00000 3726G 1868G 1857G 50.14 1.00 108 >>>> 146 hdd 3.64499 1.00000 3726G 2078G 1647G 55.77 1.11 120 >>>> 147 hdd 3.64499 1.00000 3726G 1992G 1733G 53.49 1.06 115 >>>> 148 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100 >>>> 149 hdd 3.64499 1.00000 3726G 2162G 1563G 58.03 1.15 125 >>>> 150 hdd 3.64499 1.00000 3726G 2005G 1720G 53.83 1.07 116 >>>> 151 hdd 3.64499 1.00000 3726G 1918G 1807G 51.49 1.02 111 >>>> 152 hdd 3.64499 1.00000 3726G 1921G 1804G 51.57 1.03 111 >>>> 153 hdd 3.64499 1.00000 3726G 1783G 1942G 47.86 0.95 103 >>>> 154 hdd 3.64499 1.00000 3726G 2005G 1720G 53.82 1.07 116 >>>> 155 hdd 3.64499 1.00000 3726G 1853G 1873G 49.73 0.99 107 >>>> 156 hdd 3.64499 1.00000 3726G 1626G 2099G 43.65 0.87 94 >>>> 157 hdd 3.64499 1.00000 3726G 1802G 1923G 48.38 0.96 104 >>>> 158 hdd 3.64499 1.00000 3726G 1765G 1960G 47.39 0.94 102 >>>> 159 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115 >>>> 160 hdd 3.64499 1.00000 3726G 1901G 1824G 51.02 1.02 110 >>>> 161 hdd 3.64499 1.00000 3726G 1906G 1819G 51.16 1.02 110 >>>> 162 hdd 3.64499 1.00000 3726G 1763G 1962G 47.33 0.94 102 >>>> 163 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104 >>>> 164 hdd 3.64499 1.00000 3726G 1922G 1803G 51.61 1.03 111 >>>> 165 hdd 3.64499 1.00000 3726G 1731G 1994G 46.46 0.92 100 >>>> 166 hdd 3.64499 1.00000 3726G 2027G 1699G 54.40 1.08 117 >>>> 167 hdd 3.64499 1.00000 3726G 1956G 1769G 52.50 1.04 113 >>>> 168 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107 >>>> 169 hdd 3.64499 1.00000 3726G 1884G 1841G 50.59 1.01 109 >>>> 170 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 >>>> 171 hdd 3.64499 1.00000 3726G 1898G 1827G 50.96 1.01 110 >>>> 172 hdd 3.64499 1.00000 3726G 1781G 1944G 47.82 0.95 103 >>>> 173 hdd 3.64499 1.00000 3726G 1628G 2097G 43.71 0.87 94 >>>> 174 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 >>>> 175 hdd 3.64499 1.00000 3726G 1905G 1820G 51.15 1.02 110 >>>> 176 hdd 3.64499 1.00000 3726G 1709G 2016G 45.89 0.91 99 >>>> 177 hdd 3.64499 1.00000 3726G 1991G 1734G 53.44 1.06 115 >>>> 178 hdd 3.64499 1.00000 3726G 1973G 1752G 52.97 1.05 114 >>>> 179 hdd 3.64499 1.00000 3726G 1574G 2151G 42.26 0.84 91 >>>> 180 hdd 3.64499 1.00000 3726G 1917G 1808G 51.46 1.02 111 >>>> 181 hdd 3.64499 1.00000 3726G 1937G 1788G 52.00 1.03 112 >>>> 182 hdd 3.64499 1.00000 3726G 1989G 1736G 53.40 1.06 115 >>>> 183 hdd 3.64499 1.00000 3726G 2039G 1686G 54.74 1.09 118 >>>> 184 hdd 3.64499 1.00000 3726G 1868G 1857G 50.16 1.00 108 >>>> 185 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113 >>>> 186 hdd 3.64499 1.00000 3726G 1850G 1875G 49.66 0.99 107 >>>> 187 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107 >>>> 188 hdd 3.64499 1.00000 3726G 1814G 1911G 48.69 0.97 105 >>>> 189 hdd 3.64499 1.00000 3726G 2286G 1439G 61.36 1.22 132 >>>> 190 hdd 3.64499 1.00000 3726G 2060G 1665G 55.31 1.10 119 >>>> 191 hdd 3.64499 1.00000 3726G 1836G 1889G 49.29 0.98 106 >>>> 192 hdd 3.63869 1.00000 3726G 1721G 2004G 46.21 0.92 101 >>>> 193 hdd 3.64499 1.00000 3726G 1953G 1772G 52.43 1.04 113 >>>> 194 hdd 3.64499 1.00000 3726G 1541G 2184G 41.37 0.82 89 >>>> 195 hdd 3.64499 1.00000 3726G 1837G 1888G 49.32 0.98 106 >>>> 196 hdd 3.63899 1.00000 3726G 1705G 2020G 45.79 0.91 100 >>>> 197 hdd 3.63899 1.00000 3726G 1743G 1982G 46.78 0.93 102 >>>> 198 hdd 3.63899 1.00000 3726G 1982G 1743G 53.21 1.06 116 >>>> 199 hdd 3.63899 1.00000 3726G 1382G 2343G 37.11 0.74 81 >>>> 200 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111 >>>> 201 hdd 3.63899 1.00000 3726G 1845G 1880G 49.54 0.99 108 >>>> 202 hdd 3.63899 1.00000 3726G 1658G 2067G 44.51 0.89 97 >>>> 203 hdd 3.63899 1.00000 3726G 2066G 1659G 55.47 1.10 121 >>>> 204 hdd 3.63899 1.00000 3726G 1672G 2053G 44.90 0.89 98 >>>> 205 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89 >>>> 206 hdd 3.63899 1.00000 3726G 1796G 1929G 48.22 0.96 105 >>>> 207 hdd 3.63899 1.00000 3726G 1555G 2170G 41.75 0.83 91 >>>> 208 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102 >>>> 209 hdd 3.63899 1.00000 3726G 2016G 1709G 54.11 1.08 118 >>>> 210 hdd 3.63899 1.00000 3726G 2136G 1589G 57.34 1.14 125 >>>> 211 hdd 3.63899 1.00000 3726G 1656G 2069G 44.46 0.88 97 >>>> 212 hdd 3.63899 1.00000 3726G 1553G 2172G 41.71 0.83 91 >>>> 213 hdd 3.63899 1.00000 3726G 1758G 1967G 47.19 0.94 103 >>>> 214 hdd 3.63899 1.00000 3726G 1603G 2122G 43.03 0.86 94 >>>> 215 hdd 3.63899 1.00000 3726G 1809G 1916G 48.57 0.97 106 >>>> 216 hdd 3.63899 1.00000 3726G 1760G 1965G 47.26 0.94 103 >>>> 217 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102 >>>> 218 hdd 3.63899 1.00000 3726G 1880G 1845G 50.46 1.00 110 >>>> 219 hdd 3.63899 1.00000 3726G 1693G 2032G 45.45 0.90 99 >>>> 220 hdd 3.63899 1.00000 3726G 2032G 1693G 54.56 1.09 119 >>>> 221 hdd 3.63899 1.00000 3726G 1842G 1883G 49.44 0.98 108 >>>> 222 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89 >>>> 223 hdd 3.63899 1.00000 3726G 1470G 2255G 39.46 0.79 86 >>>> 224 hdd 3.63899 1.00000 3726G 1808G 1917G 48.53 0.97 106 >>>> 225 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111 >>>> 226 hdd 3.63899 1.00000 3726G 1776G 1949G 47.69 0.95 104 >>>> 227 hdd 3.63899 1.00000 3726G 1929G 1796G 51.78 1.03 113 >>>> 228 hdd 3.63899 1.00000 3726G 1657G 2068G 44.48 0.89 97 >>>> 229 hdd 3.63899 1.00000 3726G 1843G 1882G 49.47 0.98 108 >>>> TOTAL 825T 414T 410T 50.24 >>>> MIN/MAX VAR: 0.01/1.29 STDDEV: 9.22 >>>> -------------------------------------------------------------- >>>> >>>> Am 20.10.18 um 20:35 schrieb Janne Johansson: >>>>> Yes, if you have uneven sizes I guess you could end up in a situation >>>>> where you have >>>>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces >>>>> the pool to have one >>>>> PG replica on the 1TB OSD, then it would be possible to state "this >>>>> pool cant write more than X G" >>>>> but when it is full, there would be free space left on some of the >>>>> 2TB-OSDs, but which the pool >>>>> cant utilize. Probably same for uneven OSD hosts if you have those. >>>>> >>>>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth >>>>> <freyermuth@xxxxxxxxxxxxxxxxxx>: >>>>>> >>>>>> Dear Janne, >>>>>> >>>>>> yes, of course. But since we only have two pools here, this can not explain the difference. >>>>>> The metadata is replicated (3 copies) across ssd drives, and we have < 3 TB of total raw storage for that. >>>>>> So looking at the raw space usage, we can ignore that. >>>>>> >>>>>> All the rest is used for the ceph_data pool. So the ceph_data pool, in terms of raw storage, is about 50 % used. >>>>>> >>>>>> But in terms of storage shown for that pool, it's almost 63 % %USED. >>>>>> So I guess this can purely be from bad balancing, correct? >>>>>> >>>>>> Cheers, >>>>>> Oliver >>>>>> >>>>>> Am 20.10.18 um 19:49 schrieb Janne Johansson: >>>>>>> Do mind that drives may have more than one pool on them, so RAW space >>>>>>> is what it says, how much free space there is. Then the avail and >>>>>>> %USED on per-pool stats will take replication into account, it can >>>>>>> tell how much data you may write into that particular pool, given that >>>>>>> pools replication or EC settings. >>>>>>> >>>>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth >>>>>>> <freyermuth@xxxxxxxxxxxxxxxxxx>: >>>>>>>> >>>>>>>> Dear Cephalopodians, >>>>>>>> >>>>>>>> as many others, I'm also a bit confused by "ceph df" output >>>>>>>> in a pretty straightforward configuration. >>>>>>>> >>>>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile. >>>>>>>> >>>>>>>> I get: >>>>>>>> ---------------------------------------------------------------------------- >>>>>>>> # ceph df >>>>>>>> GLOBAL: >>>>>>>> SIZE AVAIL RAW USED %RAW USED >>>>>>>> 824T 410T 414T 50.26 >>>>>>>> POOLS: >>>>>>>> NAME ID USED %USED MAX AVAIL OBJECTS >>>>>>>> cephfs_metadata 1 452M 0.05 860G 365774 >>>>>>>> cephfs_data 2 275T 62.68 164T 75056403 >>>>>>>> ---------------------------------------------------------------------------- >>>>>>>> >>>>>>>> So about 50 % of raw space are used, but already ~63 % of filesystem space are used. >>>>>>>> Is this purely from imperfect balancing? >>>>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % usage down to 37.12 %. >>>>>>>> >>>>>>>> We did not yet use the balancer plugin. >>>>>>>> We don't have any pre-luminous clients. >>>>>>>> In that setup, I take it that "upmap" mode would be recommended - correct? >>>>>>>> Any "gotchas" using that on luminous? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Oliver >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list >>>>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > >
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
