Re: RESEND: Re: PG Balancer Upmap mode not working

[Thomas Schneider <74cmonty@xxxxxxxxx>]

Hello David,

many thanks for testing the new balancer code with my OSDmap.

These are good news that balancer will "repair" my unbalanced pg
distribution on my most critical pool "hdb_backup".

However I have a follow-up question:
Currently I run
ceph osd reweight-by-utilization 105 0.1 5
3x / day via cronjob.

And this continuously modifies the REWEIGHT paramater of a specific OSD
(class HDD, size 1.6TB).

Once I enable the new balancer code, must I then reset any REWEIGHT to
1.0 for all OSDs?
If yes, when should I reset REWEIGHT? Before or after enabling balancer?
Certainly I would disable the cronjob "osd reweight-by-utilization" before.

Regards
Thomas

Am 12.12.2019 um 01:34 schrieb David Zafman:
>
> Thomas,
>
> I have a master branch version of the code to test.  The nautilus
> backport https://github.com/ceph/ceph/pull/31956 should be the same.
>
> Using your OSDMap, the code in master branch and some additional
> changes to osdmaptool I was able to balance your cluster.  The
> osdmaptool changes simulate the mgr active balancer behavior.   It
> took 302 rounds with 10 maximum upmaps per crush rule set of pools per
> round.  With the default 1 minute sleeps inside the mgr it would take
> about 5 hours.  Obviously, recovery/backfill has to finish before the
> cluster settles into the new configuration.  It needed 3402 additional
> upmaps.
>
> Because all pools for a given crush rule are balanced together you can
> see that this is more balanced than Rich's configuration using
> Luminous. The pool hdb_backup which has its own rule was the most
> difficult to balance.
>
> This balancer code is subject to change before final release of the
> next Nautilus point release.
>
>
> Final layout:
>
> osd.0 pgs 33
> osd.1 pgs 31
> osd.2 pgs 34
> osd.3 pgs 33
> osd.4 pgs 31
> osd.5 pgs 31
> osd.6 pgs 32
> osd.7 pgs 31
> osd.8 pgs 264
> osd.9 pgs 28
> osd.10 pgs 29
> osd.11 pgs 28
> osd.12 pgs 263
> osd.13 pgs 27
> osd.14 pgs 27
> osd.15 pgs 27
> osd.16 pgs 262
> osd.17 pgs 27
> osd.18 pgs 27
> osd.19 pgs 262
> osd.20 pgs 28
> osd.21 pgs 27
> osd.22 pgs 28
> osd.23 pgs 29
> osd.24 pgs 29
> osd.25 pgs 28
> osd.26 pgs 263
> osd.27 pgs 28
> osd.28 pgs 28
> osd.29 pgs 27
> osd.30 pgs 262
> osd.31 pgs 86
> osd.32 pgs 86
> osd.33 pgs 86
> osd.34 pgs 86
> osd.35 pgs 87
> osd.36 pgs 86
> osd.37 pgs 86
> osd.38 pgs 86
> osd.39 pgs 86
> osd.40 pgs 86
> osd.41 pgs 86
> osd.42 pgs 49
> osd.43 pgs 62
> osd.44 pgs 62
> osd.45 pgs 56
> osd.46 pgs 49
> osd.47 pgs 56
> osd.48 pgs 56
> osd.49 pgs 56
> osd.50 pgs 49
> osd.51 pgs 62
> osd.52 pgs 56
> osd.53 pgs 43
> osd.54 pgs 55
> osd.55 pgs 56
> osd.56 pgs 49
> osd.57 pgs 49
> osd.58 pgs 49
> osd.59 pgs 86
> osd.60 pgs 86
> osd.61 pgs 86
> osd.62 pgs 86
> osd.63 pgs 86
> osd.64 pgs 87
> osd.65 pgs 86
> osd.66 pgs 87
> osd.67 pgs 87
> osd.68 pgs 87
> osd.69 pgs 86
> osd.70 pgs 86
> osd.71 pgs 86
> osd.72 pgs 86
> osd.73 pgs 87
> osd.74 pgs 87
> osd.75 pgs 86
> osd.76 pgs 49
> osd.77 pgs 49
> osd.78 pgs 55
>
> osd.79 pgs 49
> osd.80 pgs 62
> osd.81 pgs 62
> osd.82 pgs 62
> osd.83 pgs 49
> osd.84 pgs 56
> osd.85 pgs 61
> osd.86 pgs 49
> osd.87 pgs 61
> osd.88 pgs 62
> osd.89 pgs 43
> osd.90 pgs 49
> osd.91 pgs 55
> osd.92 pgs 62
> osd.93 pgs 61
> osd.94 pgs 55
> osd.95 pgs 56
> osd.96 pgs 49
> osd.97 pgs 43
> osd.98 pgs 49
> osd.99 pgs 56
> osd.100 pgs 43
> osd.101 pgs 49
> osd.102 pgs 43
> osd.103 pgs 55
> osd.104 pgs 43
> osd.105 pgs 43
> osd.106 pgs 62
> osd.107 pgs 49
> osd.108 pgs 62
> osd.109 pgs 62
> osd.110 pgs 62
> osd.111 pgs 62
> osd.112 pgs 49
> osd.113 pgs 49
> osd.114 pgs 49
> osd.115 pgs 49
> osd.116 pgs 62
> osd.117 pgs 56
> osd.118 pgs 55
> osd.119 pgs 49
> osd.120 pgs 43
> osd.121 pgs 55
> osd.122 pgs 31
> osd.123 pgs 56
> osd.124 pgs 62
> osd.125 pgs 62
> osd.126 pgs 62
> osd.127 pgs 49
> osd.128 pgs 49
> osd.129 pgs 56
> osd.130 pgs 61
> osd.131 pgs 55
> osd.132 pgs 55
> osd.133 pgs 55
> osd.134 pgs 49
> osd.135 pgs 55
> osd.136 pgs 55
> osd.137 pgs 43
> osd.138 pgs 55
> osd.139 pgs 55
> osd.140 pgs 49
> osd.141 pgs 43
> osd.142 pgs 55
> osd.143 pgs 55
> osd.144 pgs 55
> osd.145 pgs 55
> osd.146 pgs 55
> osd.147 pgs 62
> osd.148 pgs 55
> osd.149 pgs 62
> osd.150 pgs 49
> osd.151 pgs 55
> osd.152 pgs 55
> osd.153 pgs 49
> osd.154 pgs 49
> osd.155 pgs 55
> osd.156 pgs 43
> osd.157 pgs 49
>
> osd.157 pgs 49
> osd.158 pgs 62
> osd.159 pgs 49
> osd.160 pgs 55
> osd.161 pgs 49
> osd.162 pgs 62
> osd.163 pgs 49
> osd.164 pgs 55
> osd.165 pgs 62
> osd.166 pgs 49
> osd.167 pgs 43
> osd.168 pgs 49
> osd.169 pgs 61
> osd.170 pgs 62
> osd.171 pgs 55
> osd.172 pgs 87
> osd.173 pgs 87
> osd.174 pgs 87
> osd.175 pgs 87
> osd.176 pgs 87
> osd.177 pgs 86
> osd.178 pgs 86
> osd.179 pgs 87
> osd.180 pgs 86
> osd.181 pgs 86
> osd.182 pgs 87
> osd.183 pgs 87
> osd.184 pgs 86
> osd.185 pgs 86
> osd.186 pgs 87
> osd.187 pgs 86
> osd.188 pgs 87
> osd.189 pgs 43
> osd.190 pgs 49
> osd.191 pgs 55
> osd.192 pgs 62
> osd.193 pgs 55
> osd.194 pgs 43
> osd.195 pgs 62
> osd.196 pgs 43
> osd.197 pgs 49
> osd.198 pgs 62
> osd.199 pgs 55
> osd.200 pgs 37
> osd.201 pgs 55
> osd.202 pgs 49
> osd.203 pgs 49
> osd.204 pgs 62
> osd.205 pgs 62
> osd.206 pgs 55
> osd.207 pgs 49
> osd.208 pgs 43
> osd.209 pgs 43
> osd.210 pgs 43
> osd.211 pgs 49
> osd.212 pgs 49
> osd.213 pgs 49
> osd.214 pgs 55
> osd.215 pgs 43
> osd.216 pgs 49
> osd.217 pgs 49
> osd.218 pgs 55
> osd.219 pgs 62
> osd.220 pgs 49
> osd.221 pgs 55
> osd.222 pgs 49
> osd.223 pgs 55
> osd.224 pgs 49
> osd.225 pgs 55
> osd.226 pgs 49
> osd.227 pgs 55
> osd.228 pgs 62
> osd.229 pgs 62
> osd.230 pgs 37
> osd.231 pgs 55
> osd.232 pgs 62
> osd.233 pgs 55
> osd.234 pgs 55
> osd.235 pgs 62
>
> osd.236 pgs 55
> osd.237 pgs 62
> osd.238 pgs 61
> osd.239 pgs 37
> osd.240 pgs 62
> osd.241 pgs 62
> osd.242 pgs 61
> osd.243 pgs 62
> osd.244 pgs 62
> osd.245 pgs 62
> osd.246 pgs 62
> osd.247 pgs 55
> osd.248 pgs 62
> osd.249 pgs 49
> osd.250 pgs 61
> osd.251 pgs 49
> osd.252 pgs 43
> osd.253 pgs 55
> osd.254 pgs 49
> osd.255 pgs 55
> osd.256 pgs 43
> osd.257 pgs 55
> osd.258 pgs 62
> osd.259 pgs 61
> osd.260 pgs 55
> osd.261 pgs 43
> osd.262 pgs 43
> osd.263 pgs 49
> osd.264 pgs 49
> osd.265 pgs 62
> osd.266 pgs 49
> osd.267 pgs 62
> osd.268 pgs 87
> osd.269 pgs 86
> osd.270 pgs 87
> osd.271 pgs 86
> osd.272 pgs 270
> osd.273 pgs 271
> osd.274 pgs 270
> osd.275 pgs 270
> osd.276 pgs 270
> osd.277 pgs 270
> osd.278 pgs 270
> osd.279 pgs 270
> osd.280 pgs 270
> osd.281 pgs 270
> osd.282 pgs 270
> osd.283 pgs 271
> osd.284 pgs 270
> osd.285 pgs 271
> osd.286 pgs 270
> osd.287 pgs 270
> osd.288 pgs 270
> osd.289 pgs 270
> osd.290 pgs 270
> osd.291 pgs 270
> osd.292 pgs 270
> osd.293 pgs 270
> osd.294 pgs 270
> osd.295 pgs 270
> osd.296 pgs 270
> osd.297 pgs 270
> osd.298 pgs 270
> osd.299 pgs 270
> osd.300 pgs 270
> osd.301 pgs 270
> osd.302 pgs 270
> osd.303 pgs 270
> osd.304 pgs 270
> osd.305 pgs 270
> osd.306 pgs 270
> osd.307 pgs 270
> osd.308 pgs 270
> osd.309 pgs 270
> osd.310 pgs 270
> osd.311 pgs 270
> osd.312 pgs 270
> osd.313 pgs 270
>
> osd.314 pgs 270
> osd.315 pgs 270
> osd.316 pgs 270
> osd.317 pgs 270
> osd.318 pgs 270
> osd.319 pgs 270
> osd.320 pgs 271
> osd.321 pgs 271
> osd.322 pgs 271
> osd.323 pgs 271
> osd.324 pgs 271
> osd.325 pgs 270
> osd.326 pgs 270
> osd.327 pgs 270
> osd.328 pgs 270
> osd.329 pgs 270
> osd.330 pgs 270
> osd.331 pgs 270
> osd.332 pgs 271
> osd.333 pgs 270
> osd.334 pgs 271
> osd.335 pgs 270
> osd.336 pgs 270
> osd.337 pgs 271
> osd.338 pgs 271
> osd.339 pgs 270
> osd.340 pgs 270
> osd.341 pgs 270
> osd.342 pgs 270
> osd.343 pgs 270
> osd.344 pgs 270
> osd.345 pgs 271
> osd.346 pgs 270
> osd.347 pgs 271
> osd.348 pgs 270
> osd.349 pgs 270
> osd.350 pgs 270
> osd.351 pgs 270
> osd.352 pgs 270
> osd.353 pgs 270
> osd.354 pgs 270
> osd.355 pgs 270

> osd.356 pgs 270
> osd.357 pgs 270
> osd.358 pgs 270
> osd.359 pgs 270
> osd.360 pgs 270
> osd.361 pgs 270
> osd.362 pgs 270
> osd.363 pgs 270
> osd.364 pgs 270
> osd.365 pgs 270
> osd.366 pgs 270
> osd.367 pgs 270
> osd.368 pgs 86
> osd.369 pgs 86
> osd.370 pgs 87
> osd.371 pgs 270
> osd.372 pgs 270
> osd.373 pgs 270
> osd.374 pgs 270
> osd.375 pgs 270
> osd.376 pgs 270
> osd.377 pgs 270
> osd.378 pgs 270
> osd.379 pgs 270
> osd.380 pgs 270
> osd.381 pgs 270
> osd.382 pgs 270
> osd.383 pgs 270
> osd.384 pgs 271
> osd.385 pgs 270
> osd.386 pgs 270
> osd.387 pgs 270
> osd.388 pgs 270
> osd.389 pgs 270
> osd.390 pgs 270
> osd.391 pgs 270
>
> osd.392 pgs 270
> osd.393 pgs 270
> osd.394 pgs 270
> osd.395 pgs 270
> osd.396 pgs 270
> osd.397 pgs 270
> osd.398 pgs 270
> osd.399 pgs 270
> osd.400 pgs 270
> osd.401 pgs 270
> osd.402 pgs 270
> osd.403 pgs 270
> osd.404 pgs 270
> osd.405 pgs 270
> osd.406 pgs 271
> osd.407 pgs 270
> osd.408 pgs 270
> osd.409 pgs 270
> osd.410 pgs 270
> osd.411 pgs 270
> osd.412 pgs 270
> osd.413 pgs 270
> osd.414 pgs 270
> osd.415 pgs 270
> osd.416 pgs 270
> osd.417 pgs 270
> osd.418 pgs 270
> osd.419 pgs 87
> osd.420 pgs 87
> osd.421 pgs 86
> osd.422 pgs 86
> osd.423 pgs 86
> osd.424 pgs 86
> osd.425 pgs 86
> osd.426 pgs 86
> osd.427 pgs 86
> osd.428 pgs 86
> osd.429 pgs 86
> osd.430 pgs 86
> osd.431 pgs 86
> osd.432 pgs 86
> osd.433 pgs 86
> osd.434 pgs 86
> osd.435 pgs 58
> osd.436 pgs 58
> osd.437 pgs 58
> osd.438 pgs 58
> osd.439 pgs 58
> osd.440 pgs 58
> osd.441 pgs 58
>
>
> David
>
> On 12/11/19 6:12 AM, Thomas Schneider wrote:
>> Hello David,
>>
>> I'm experiencing issues with OSD balancing, too.
>> My ceph cluster is running on release
>> ceph version 14.2.4.1 (596a387fb278758406deabf997735a1f706660c9)
>> nautilus (stable)
>>
>> Would you be able to test (the latest code) on my OSDmap and verify if
>> balancing would work?
>> I have attached it to this email.
>>
>> Regards
>> Thomas
>>
>>
>

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx