I've tried upmap balancer but pg distribution was not good. I think Crush-compat is way better with Nautilus and also its the default option. As you can see at df tree pg distribution is not that bad. Also max avail never changed. If its not the reason for max avail then what it is? Maybe I should create a new pool and check its max avail. If the New pool max avail greater then other pools then ceph recalculation is bugy or need to triger somehow. 1 Eyl 2021 Çar 17:07 tarihinde Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx> şunu yazdı: > Googling for that balancer error message, I came across > https://tracker.ceph.com/issues/22814, which was closed/wont-fix, and > some threads that claimed that class-based crush rules actually use > some form of shadow trees in the background. I'm not sure how accurate > that is. > > The only suggestion I have, which is what was also suggested in one of > the above threads, is to use the upmap balancer instead if possible. > > Josh > > On Wed, Sep 1, 2021 at 2:38 AM mhnx <morphinwithyou@xxxxxxxxx> wrote: > > > > ceph osd crush tree (I only have one subtree and its root default) > > ID CLASS WEIGHT (compat) TYPE NAME > > -1 2785.87891 root default > > -3 280.04803 280.04803 host NODE-1 > > 0 hdd 14.60149 14.60149 osd.0 > > 19 ssd 0.87320 0.87320 osd.19 > > 208 ssd 0.87329 0.87329 osd.208 > > 209 ssd 0.87329 0.87329 osd.209 > > -7 280.04803 280.04803 host NODE-2 > > 38 hdd 14.60149 14.60149 osd.38 > > 39 ssd 0.87320 0.87320 osd.39 > > 207 ssd 0.87329 0.87329 osd.207 > > 210 ssd 0.87329 0.87329 osd.210 > > -10 280.04803 280.04803 host NODE-3 > > 58 hdd 14.60149 14.60149 osd.58 > > 59 ssd 0.87320 0.87320 osd.59 > > 203 ssd 0.87329 0.87329 osd.203 > > 211 ssd 0.87329 0.87329 osd.211 > > -13 280.04803 280.04803 host NODE-4 > > 78 hdd 14.60149 14.60149 osd.78 > > 79 ssd 0.87320 0.87320 osd.79 > > 206 ssd 0.87329 0.87329 osd.206 > > 212 ssd 0.87329 0.87329 osd.212 > > -16 280.04803 280.04803 host NODE-5 > > 98 hdd 14.60149 14.60149 osd.98 > > 99 ssd 0.87320 0.87320 osd.99 > > 205 ssd 0.87329 0.87329 osd.205 > > 213 ssd 0.87329 0.87329 osd.213 > > -19 265.44662 265.44662 host NODE-6 > > 118 hdd 14.60149 14.60149 osd.118 > > 114 ssd 0.87329 0.87329 osd.114 > > 200 ssd 0.87329 0.87329 osd.200 > > 214 ssd 0.87329 0.87329 osd.214 > > -22 280.04803 280.04803 host NODE-7 > > 138 hdd 14.60149 14.60149 osd.138 > > 139 ssd 0.87320 0.87320 osd.139 > > 204 ssd 0.87329 0.87329 osd.204 > > 215 ssd 0.87329 0.87329 osd.215 > > -25 280.04810 280.04810 host NODE-8 > > 158 hdd 14.60149 14.60149 osd.158 > > 119 ssd 0.87329 0.87329 osd.119 > > 159 ssd 0.87329 0.87329 osd.159 > > 216 ssd 0.87329 0.87329 osd.216 > > -28 280.04810 280.04810 host NODE-9 > > 178 hdd 14.60149 14.60149 osd.178 > > 179 ssd 0.87329 0.87329 osd.179 > > 201 ssd 0.87329 0.87329 osd.201 > > 217 ssd 0.87329 0.87329 osd.217 > > -31 280.04803 280.04803 host NODE-10 > > 180 hdd 14.60149 14.60149 osd.180 > > 199 ssd 0.87320 0.87320 osd.199 > > 202 ssd 0.87329 0.87329 osd.202 > > 218 ssd 0.87329 0.87329 osd.218 > > > > This pg "6.dc" is on 199,213,217 OSD's. > > > > 6.dc 812 0 0 0 0 > 1369675264 0 0 3005 3005 > active+clean 2021-08-31 16:36:06.645208 32265'415965 32265:287175109 > [199,213,217] > > > > ceph osd df tree | grep "CLASS\|ssd" | grep ".199\|.213\|217" > > 199 ssd 0.87320 1.00000 894 GiB 281 GiB 119 GiB 159 GiB 2.5 GiB > 614 GiB 31.38 0.52 103 up osd.199 > > 213 ssd 0.87329 1.00000 894 GiB 291 GiB 95 GiB 195 GiB 2.3 GiB > 603 GiB 32.59 0.54 95 up osd.213 > > 217 ssd 0.87329 1.00000 894 GiB 261 GiB 83 GiB 176 GiB 2.3 GiB > 633 GiB 29.18 0.48 89 up osd.217 > > > > As you can see the pg lives on 3 ssd OSD's and one of them is the new > one. So we can not say it belongs to someone else. > > > > rule ssd-rule { > > id 1 > > type replicated > > step take default class ssd > > step chooseleaf firstn 0 type host > > step emit > > } > > > > pool 54 'rgw.buckets.index' replicated size 3 min_size 1 crush_rule 1 > object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change > 31607 lfor 0/0/30823 flags hashpspool stripe_width 0 compression_algorithm > lz4 compression_mode aggressive application rgw > > > > What is the next step? > > > > > > Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>, 1 Eyl 2021 Çar, 04:03 > tarihinde şunu yazdı: > >> > >> Yeah, I would suggest inspecting your CRUSH tree. Unfortunately the > >> grep above removed that information from 'df tree', but from the > >> information you provided there does appear to be a significant > >> imbalance remaining. > >> > >> Josh > >> > >> On Tue, Aug 31, 2021 at 6:02 PM mhnx <morphinwithyou@xxxxxxxxx> wrote: > >> > > >> > Hello Josh! > >> > > >> > I use balancer active - crush-compat. Balance is done and there are > no remapped pgs at ceph -s > >> > > >> > ceph osd df tree | grep 'CLASS\|ssd' > >> > > >> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS TYPE NAME > >> > 19 ssd 0.87320 1.00000 894 GiB 402 GiB 117 GiB 281 GiB 3.0 GiB > 492 GiB 44.93 0.74 102 up osd.19 > >> > 208 ssd 0.87329 1.00000 894 GiB 205 GiB 85 GiB 113 GiB 6.6 GiB > 690 GiB 22.89 0.38 95 up osd.208 > >> > 209 ssd 0.87329 1.00000 894 GiB 204 GiB 87 GiB 114 GiB 2.7 GiB > 690 GiB 22.84 0.38 65 up osd.209 > >> > 199 ssd 0.87320 1.00000 894 GiB 281 GiB 118 GiB 159 GiB 2.8 GiB > 614 GiB 31.37 0.52 103 up osd.199 > >> > 202 ssd 0.87329 1.00000 894 GiB 278 GiB 89 GiB 183 GiB 6.3 GiB > 616 GiB 31.08 0.51 97 up osd.202 > >> > 218 ssd 0.87329 1.00000 894 GiB 201 GiB 75 GiB 124 GiB 1.8 GiB > 693 GiB 22.46 0.37 84 up osd.218 > >> > 39 ssd 0.87320 1.00000 894 GiB 334 GiB 86 GiB 242 GiB 5.3 GiB > 560 GiB 37.34 0.61 91 up osd.39 > >> > 207 ssd 0.87329 1.00000 894 GiB 232 GiB 88 GiB 138 GiB 7.0 GiB > 662 GiB 25.99 0.43 81 up osd.207 > >> > 210 ssd 0.87329 1.00000 894 GiB 270 GiB 109 GiB 160 GiB 1.4 GiB > 624 GiB 30.18 0.50 99 up osd.210 > >> > 59 ssd 0.87320 1.00000 894 GiB 374 GiB 127 GiB 244 GiB 3.1 GiB > 520 GiB 41.79 0.69 97 up osd.59 > >> > 203 ssd 0.87329 1.00000 894 GiB 314 GiB 96 GiB 210 GiB 7.5 GiB > 581 GiB 35.06 0.58 104 up osd.203 > >> > 211 ssd 0.87329 1.00000 894 GiB 231 GiB 60 GiB 169 GiB 1.7 GiB > 663 GiB 25.82 0.42 81 up osd.211 > >> > 79 ssd 0.87320 1.00000 894 GiB 409 GiB 109 GiB 298 GiB 2.0 GiB > 486 GiB 45.70 0.75 102 up osd.79 > >> > 206 ssd 0.87329 1.00000 894 GiB 284 GiB 107 GiB 175 GiB 1.9 GiB > 610 GiB 31.79 0.52 94 up osd.206 > >> > 212 ssd 0.87329 1.00000 894 GiB 239 GiB 85 GiB 152 GiB 2.0 GiB > 655 GiB 26.71 0.44 80 up osd.212 > >> > 99 ssd 0.87320 1.00000 894 GiB 392 GiB 73 GiB 314 GiB 4.7 GiB > 503 GiB 43.79 0.72 85 up osd.99 > >> > 205 ssd 0.87329 1.00000 894 GiB 445 GiB 87 GiB 353 GiB 4.8 GiB > 449 GiB 49.80 0.82 95 up osd.205 > >> > 213 ssd 0.87329 1.00000 894 GiB 291 GiB 94 GiB 194 GiB 2.3 GiB > 603 GiB 32.57 0.54 95 up osd.213 > >> > 114 ssd 0.87329 1.00000 894 GiB 319 GiB 125 GiB 191 GiB 3.0 GiB > 575 GiB 35.67 0.59 99 up osd.114 > >> > 200 ssd 0.87329 1.00000 894 GiB 231 GiB 78 GiB 150 GiB 2.9 GiB > 663 GiB 25.83 0.42 90 up osd.200 > >> > 214 ssd 0.87329 1.00000 894 GiB 296 GiB 106 GiB 187 GiB 2.6 GiB > 598 GiB 33.09 0.54 100 up osd.214 > >> > 139 ssd 0.87320 1.00000 894 GiB 270 GiB 98 GiB 169 GiB 2.3 GiB > 624 GiB 30.18 0.50 96 up osd.139 > >> > 204 ssd 0.87329 1.00000 894 GiB 301 GiB 117 GiB 181 GiB 2.9 GiB > 593 GiB 33.64 0.55 104 up osd.204 > >> > 215 ssd 0.87329 1.00000 894 GiB 203 GiB 78 GiB 122 GiB 3.3 GiB > 691 GiB 22.69 0.37 81 up osd.215 > >> > 119 ssd 0.87329 1.00000 894 GiB 200 GiB 106 GiB 92 GiB 2.0 GiB > 694 GiB 22.39 0.37 99 up osd.119 > >> > 159 ssd 0.87329 1.00000 894 GiB 213 GiB 96 GiB 113 GiB 3.2 GiB > 682 GiB 23.77 0.39 93 up osd.159 > >> > 216 ssd 0.87329 1.00000 894 GiB 322 GiB 109 GiB 211 GiB 1.8 GiB > 573 GiB 35.96 0.59 101 up osd.216 > >> > 179 ssd 0.87329 1.00000 894 GiB 389 GiB 85 GiB 300 GiB 3.2 GiB > 505 GiB 43.49 0.71 104 up osd.179 > >> > 201 ssd 0.87329 1.00000 894 GiB 494 GiB 104 GiB 386 GiB 4.1 GiB > 401 GiB 55.20 0.91 103 up osd.201 > >> > 217 ssd 0.87329 1.00000 894 GiB 261 GiB 83 GiB 176 GiB 2.3 GiB > 634 GiB 29.15 0.48 89 up osd.217 > >> > > >> > > >> > When I check the balancer status I saw that: ""optimize_result": > "Some osds belong to multiple subtrees:" > >> > Do I need to check crushmap? > >> > > >> > > >> > > >> > Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>, 31 Ağu 2021 Sal, 22:32 > tarihinde şunu yazdı: > >> >> > >> >> Hi there, > >> >> > >> >> Could you post the output of "ceph osd df tree"? I would highly > >> >> suspect that this is a result of imbalance, and that's the easiest > way > >> >> to see if that's the case. It would also confirm that the new disks > >> >> have taken on PGs. > >> >> > >> >> Josh > >> >> > >> >> On Tue, Aug 31, 2021 at 10:50 AM mhnx <morphinwithyou@xxxxxxxxx> > wrote: > >> >> > > >> >> > I'm using Nautilus 14.2.16 > >> >> > > >> >> > I was have 20 ssd OSD in my cluster and I added 10 more. " Each > SSD=960GB" > >> >> > The Size increased to *(26TiB)* as expected but the Replicated (3) > Pool Max > >> >> > Avail didn't changed *(3.5TiB)*. > >> >> > I've increased pg_num and PG rebalance is also done. > >> >> > > >> >> > Do I need any special treatment to expand the pool Max Avail? > >> >> > > >> >> > CLASS SIZE AVAIL USED RAW USED %RAW > USED > >> >> > hdd 2.7 PiB 1.0 PiB 1.6 PiB 1.6 PiB > 61.12 > >> >> > ssd *26 TiB* 18 TiB 2.8 TiB 8.7 TiB > 33.11 > >> >> > TOTAL 2.7 PiB 1.1 PiB 1.6 PiB 1.7 PiB > 60.85 > >> >> > > >> >> > POOLS: > >> >> > POOL ID PGS STORED OBJECTS > >> >> > USED %USED MAX AVAIL > >> >> > xxx.rgw.buckets.index 54 128 541 GiB > 435.69k 541 > >> >> > GiB 4.82 *3.5 TiB* > >> >> > _______________________________________________ > >> >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |