Yeah, I would suggest inspecting your CRUSH tree. Unfortunately the grep above removed that information from 'df tree', but from the information you provided there does appear to be a significant imbalance remaining. Josh On Tue, Aug 31, 2021 at 6:02 PM mhnx <morphinwithyou@xxxxxxxxx> wrote: > > Hello Josh! > > I use balancer active - crush-compat. Balance is done and there are no remapped pgs at ceph -s > > ceph osd df tree | grep 'CLASS\|ssd' > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME > 19 ssd 0.87320 1.00000 894 GiB 402 GiB 117 GiB 281 GiB 3.0 GiB 492 GiB 44.93 0.74 102 up osd.19 > 208 ssd 0.87329 1.00000 894 GiB 205 GiB 85 GiB 113 GiB 6.6 GiB 690 GiB 22.89 0.38 95 up osd.208 > 209 ssd 0.87329 1.00000 894 GiB 204 GiB 87 GiB 114 GiB 2.7 GiB 690 GiB 22.84 0.38 65 up osd.209 > 199 ssd 0.87320 1.00000 894 GiB 281 GiB 118 GiB 159 GiB 2.8 GiB 614 GiB 31.37 0.52 103 up osd.199 > 202 ssd 0.87329 1.00000 894 GiB 278 GiB 89 GiB 183 GiB 6.3 GiB 616 GiB 31.08 0.51 97 up osd.202 > 218 ssd 0.87329 1.00000 894 GiB 201 GiB 75 GiB 124 GiB 1.8 GiB 693 GiB 22.46 0.37 84 up osd.218 > 39 ssd 0.87320 1.00000 894 GiB 334 GiB 86 GiB 242 GiB 5.3 GiB 560 GiB 37.34 0.61 91 up osd.39 > 207 ssd 0.87329 1.00000 894 GiB 232 GiB 88 GiB 138 GiB 7.0 GiB 662 GiB 25.99 0.43 81 up osd.207 > 210 ssd 0.87329 1.00000 894 GiB 270 GiB 109 GiB 160 GiB 1.4 GiB 624 GiB 30.18 0.50 99 up osd.210 > 59 ssd 0.87320 1.00000 894 GiB 374 GiB 127 GiB 244 GiB 3.1 GiB 520 GiB 41.79 0.69 97 up osd.59 > 203 ssd 0.87329 1.00000 894 GiB 314 GiB 96 GiB 210 GiB 7.5 GiB 581 GiB 35.06 0.58 104 up osd.203 > 211 ssd 0.87329 1.00000 894 GiB 231 GiB 60 GiB 169 GiB 1.7 GiB 663 GiB 25.82 0.42 81 up osd.211 > 79 ssd 0.87320 1.00000 894 GiB 409 GiB 109 GiB 298 GiB 2.0 GiB 486 GiB 45.70 0.75 102 up osd.79 > 206 ssd 0.87329 1.00000 894 GiB 284 GiB 107 GiB 175 GiB 1.9 GiB 610 GiB 31.79 0.52 94 up osd.206 > 212 ssd 0.87329 1.00000 894 GiB 239 GiB 85 GiB 152 GiB 2.0 GiB 655 GiB 26.71 0.44 80 up osd.212 > 99 ssd 0.87320 1.00000 894 GiB 392 GiB 73 GiB 314 GiB 4.7 GiB 503 GiB 43.79 0.72 85 up osd.99 > 205 ssd 0.87329 1.00000 894 GiB 445 GiB 87 GiB 353 GiB 4.8 GiB 449 GiB 49.80 0.82 95 up osd.205 > 213 ssd 0.87329 1.00000 894 GiB 291 GiB 94 GiB 194 GiB 2.3 GiB 603 GiB 32.57 0.54 95 up osd.213 > 114 ssd 0.87329 1.00000 894 GiB 319 GiB 125 GiB 191 GiB 3.0 GiB 575 GiB 35.67 0.59 99 up osd.114 > 200 ssd 0.87329 1.00000 894 GiB 231 GiB 78 GiB 150 GiB 2.9 GiB 663 GiB 25.83 0.42 90 up osd.200 > 214 ssd 0.87329 1.00000 894 GiB 296 GiB 106 GiB 187 GiB 2.6 GiB 598 GiB 33.09 0.54 100 up osd.214 > 139 ssd 0.87320 1.00000 894 GiB 270 GiB 98 GiB 169 GiB 2.3 GiB 624 GiB 30.18 0.50 96 up osd.139 > 204 ssd 0.87329 1.00000 894 GiB 301 GiB 117 GiB 181 GiB 2.9 GiB 593 GiB 33.64 0.55 104 up osd.204 > 215 ssd 0.87329 1.00000 894 GiB 203 GiB 78 GiB 122 GiB 3.3 GiB 691 GiB 22.69 0.37 81 up osd.215 > 119 ssd 0.87329 1.00000 894 GiB 200 GiB 106 GiB 92 GiB 2.0 GiB 694 GiB 22.39 0.37 99 up osd.119 > 159 ssd 0.87329 1.00000 894 GiB 213 GiB 96 GiB 113 GiB 3.2 GiB 682 GiB 23.77 0.39 93 up osd.159 > 216 ssd 0.87329 1.00000 894 GiB 322 GiB 109 GiB 211 GiB 1.8 GiB 573 GiB 35.96 0.59 101 up osd.216 > 179 ssd 0.87329 1.00000 894 GiB 389 GiB 85 GiB 300 GiB 3.2 GiB 505 GiB 43.49 0.71 104 up osd.179 > 201 ssd 0.87329 1.00000 894 GiB 494 GiB 104 GiB 386 GiB 4.1 GiB 401 GiB 55.20 0.91 103 up osd.201 > 217 ssd 0.87329 1.00000 894 GiB 261 GiB 83 GiB 176 GiB 2.3 GiB 634 GiB 29.15 0.48 89 up osd.217 > > > When I check the balancer status I saw that: ""optimize_result": "Some osds belong to multiple subtrees:" > Do I need to check crushmap? > > > > Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>, 31 Ağu 2021 Sal, 22:32 tarihinde şunu yazdı: >> >> Hi there, >> >> Could you post the output of "ceph osd df tree"? I would highly >> suspect that this is a result of imbalance, and that's the easiest way >> to see if that's the case. It would also confirm that the new disks >> have taken on PGs. >> >> Josh >> >> On Tue, Aug 31, 2021 at 10:50 AM mhnx <morphinwithyou@xxxxxxxxx> wrote: >> > >> > I'm using Nautilus 14.2.16 >> > >> > I was have 20 ssd OSD in my cluster and I added 10 more. " Each SSD=960GB" >> > The Size increased to *(26TiB)* as expected but the Replicated (3) Pool Max >> > Avail didn't changed *(3.5TiB)*. >> > I've increased pg_num and PG rebalance is also done. >> > >> > Do I need any special treatment to expand the pool Max Avail? >> > >> > CLASS SIZE AVAIL USED RAW USED %RAW USED >> > hdd 2.7 PiB 1.0 PiB 1.6 PiB 1.6 PiB 61.12 >> > ssd *26 TiB* 18 TiB 2.8 TiB 8.7 TiB 33.11 >> > TOTAL 2.7 PiB 1.1 PiB 1.6 PiB 1.7 PiB 60.85 >> > >> > POOLS: >> > POOL ID PGS STORED OBJECTS >> > USED %USED MAX AVAIL >> > xxx.rgw.buckets.index 54 128 541 GiB 435.69k 541 >> > GiB 4.82 *3.5 TiB* >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |