Hello Josh! I use balancer active - crush-compat. Balance is done and there are no remapped pgs at ceph -s ceph osd df tree | grep 'CLASS\|ssd' ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 19 ssd 0.87320 1.00000 894 GiB 402 GiB 117 GiB 281 GiB 3.0 GiB 492 GiB 44.93 0.74 102 up osd.19 208 ssd 0.87329 1.00000 894 GiB 205 GiB 85 GiB 113 GiB 6.6 GiB 690 GiB 22.89 0.38 95 up osd.208 209 ssd 0.87329 1.00000 894 GiB 204 GiB 87 GiB 114 GiB 2.7 GiB 690 GiB 22.84 0.38 65 up osd.209 199 ssd 0.87320 1.00000 894 GiB 281 GiB 118 GiB 159 GiB 2.8 GiB 614 GiB 31.37 0.52 103 up osd.199 202 ssd 0.87329 1.00000 894 GiB 278 GiB 89 GiB 183 GiB 6.3 GiB 616 GiB 31.08 0.51 97 up osd.202 218 ssd 0.87329 1.00000 894 GiB 201 GiB 75 GiB 124 GiB 1.8 GiB 693 GiB 22.46 0.37 84 up osd.218 39 ssd 0.87320 1.00000 894 GiB 334 GiB 86 GiB 242 GiB 5.3 GiB 560 GiB 37.34 0.61 91 up osd.39 207 ssd 0.87329 1.00000 894 GiB 232 GiB 88 GiB 138 GiB 7.0 GiB 662 GiB 25.99 0.43 81 up osd.207 210 ssd 0.87329 1.00000 894 GiB 270 GiB 109 GiB 160 GiB 1.4 GiB 624 GiB 30.18 0.50 99 up osd.210 59 ssd 0.87320 1.00000 894 GiB 374 GiB 127 GiB 244 GiB 3.1 GiB 520 GiB 41.79 0.69 97 up osd.59 203 ssd 0.87329 1.00000 894 GiB 314 GiB 96 GiB 210 GiB 7.5 GiB 581 GiB 35.06 0.58 104 up osd.203 211 ssd 0.87329 1.00000 894 GiB 231 GiB 60 GiB 169 GiB 1.7 GiB 663 GiB 25.82 0.42 81 up osd.211 79 ssd 0.87320 1.00000 894 GiB 409 GiB 109 GiB 298 GiB 2.0 GiB 486 GiB 45.70 0.75 102 up osd.79 206 ssd 0.87329 1.00000 894 GiB 284 GiB 107 GiB 175 GiB 1.9 GiB 610 GiB 31.79 0.52 94 up osd.206 212 ssd 0.87329 1.00000 894 GiB 239 GiB 85 GiB 152 GiB 2.0 GiB 655 GiB 26.71 0.44 80 up osd.212 99 ssd 0.87320 1.00000 894 GiB 392 GiB 73 GiB 314 GiB 4.7 GiB 503 GiB 43.79 0.72 85 up osd.99 205 ssd 0.87329 1.00000 894 GiB 445 GiB 87 GiB 353 GiB 4.8 GiB 449 GiB 49.80 0.82 95 up osd.205 213 ssd 0.87329 1.00000 894 GiB 291 GiB 94 GiB 194 GiB 2.3 GiB 603 GiB 32.57 0.54 95 up osd.213 114 ssd 0.87329 1.00000 894 GiB 319 GiB 125 GiB 191 GiB 3.0 GiB 575 GiB 35.67 0.59 99 up osd.114 200 ssd 0.87329 1.00000 894 GiB 231 GiB 78 GiB 150 GiB 2.9 GiB 663 GiB 25.83 0.42 90 up osd.200 214 ssd 0.87329 1.00000 894 GiB 296 GiB 106 GiB 187 GiB 2.6 GiB 598 GiB 33.09 0.54 100 up osd.214 139 ssd 0.87320 1.00000 894 GiB 270 GiB 98 GiB 169 GiB 2.3 GiB 624 GiB 30.18 0.50 96 up osd.139 204 ssd 0.87329 1.00000 894 GiB 301 GiB 117 GiB 181 GiB 2.9 GiB 593 GiB 33.64 0.55 104 up osd.204 215 ssd 0.87329 1.00000 894 GiB 203 GiB 78 GiB 122 GiB 3.3 GiB 691 GiB 22.69 0.37 81 up osd.215 119 ssd 0.87329 1.00000 894 GiB 200 GiB 106 GiB 92 GiB 2.0 GiB 694 GiB 22.39 0.37 99 up osd.119 159 ssd 0.87329 1.00000 894 GiB 213 GiB 96 GiB 113 GiB 3.2 GiB 682 GiB 23.77 0.39 93 up osd.159 216 ssd 0.87329 1.00000 894 GiB 322 GiB 109 GiB 211 GiB 1.8 GiB 573 GiB 35.96 0.59 101 up osd.216 179 ssd 0.87329 1.00000 894 GiB 389 GiB 85 GiB 300 GiB 3.2 GiB 505 GiB 43.49 0.71 104 up osd.179 201 ssd 0.87329 1.00000 894 GiB 494 GiB 104 GiB 386 GiB 4.1 GiB 401 GiB 55.20 0.91 103 up osd.201 217 ssd 0.87329 1.00000 894 GiB 261 GiB 83 GiB 176 GiB 2.3 GiB 634 GiB 29.15 0.48 89 up osd.217 When I check the balancer status I saw that: ""optimize_result": "Some osds belong to multiple subtrees:" Do I need to check crushmap? Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>, 31 Ağu 2021 Sal, 22:32 tarihinde şunu yazdı: > Hi there, > > Could you post the output of "ceph osd df tree"? I would highly > suspect that this is a result of imbalance, and that's the easiest way > to see if that's the case. It would also confirm that the new disks > have taken on PGs. > > Josh > > On Tue, Aug 31, 2021 at 10:50 AM mhnx <morphinwithyou@xxxxxxxxx> wrote: > > > > I'm using Nautilus 14.2.16 > > > > I was have 20 ssd OSD in my cluster and I added 10 more. " Each > SSD=960GB" > > The Size increased to *(26TiB)* as expected but the Replicated (3) Pool > Max > > Avail didn't changed *(3.5TiB)*. > > I've increased pg_num and PG rebalance is also done. > > > > Do I need any special treatment to expand the pool Max Avail? > > > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 2.7 PiB 1.0 PiB 1.6 PiB 1.6 PiB 61.12 > > ssd *26 TiB* 18 TiB 2.8 TiB 8.7 TiB > 33.11 > > TOTAL 2.7 PiB 1.1 PiB 1.6 PiB 1.7 PiB 60.85 > > > > POOLS: > > POOL ID PGS STORED OBJECTS > > USED %USED MAX AVAIL > > xxx.rgw.buckets.index 54 128 541 GiB 435.69k > 541 > > GiB 4.82 *3.5 TiB* > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |