*ceph osd crush tree (*I only have one subtree and its root default*)*
ID CLASS WEIGHT (compat) TYPE NAME
-1 2785.87891 root default
-3 280.04803 280.04803 host NODE-1
0 hdd 14.60149 14.60149 osd.0
19 ssd 0.87320 0.87320 osd.19
208 ssd 0.87329 0.87329 osd.208
209 ssd 0.87329 0.87329 osd.209
-7 280.04803 280.04803 host NODE-2
38 hdd 14.60149 14.60149 osd.38
39 ssd 0.87320 0.87320 osd.39
207 ssd 0.87329 0.87329 osd.207
210 ssd 0.87329 0.87329 osd.210
-10 280.04803 280.04803 host NODE-3
58 hdd 14.60149 14.60149 osd.58
59 ssd 0.87320 0.87320 osd.59
203 ssd 0.87329 0.87329 osd.203
211 ssd 0.87329 0.87329 osd.211
-13 280.04803 280.04803 host NODE-4
78 hdd 14.60149 14.60149 osd.78
79 ssd 0.87320 0.87320 osd.79
206 ssd 0.87329 0.87329 osd.206
212 ssd 0.87329 0.87329 osd.212
-16 280.04803 280.04803 host NODE-5
98 hdd 14.60149 14.60149 osd.98
99 ssd 0.87320 0.87320 osd.99
205 ssd 0.87329 0.87329 osd.205
213 ssd 0.87329 0.87329 osd.213
-19 265.44662 265.44662 host NODE-6
118 hdd 14.60149 14.60149 osd.118
114 ssd 0.87329 0.87329 osd.114
200 ssd 0.87329 0.87329 osd.200
214 ssd 0.87329 0.87329 osd.214
-22 280.04803 280.04803 host NODE-7
138 hdd 14.60149 14.60149 osd.138
139 ssd 0.87320 0.87320 osd.139
204 ssd 0.87329 0.87329 osd.204
215 ssd 0.87329 0.87329 osd.215
-25 280.04810 280.04810 host NODE-8
158 hdd 14.60149 14.60149 osd.158
119 ssd 0.87329 0.87329 osd.119
159 ssd 0.87329 0.87329 osd.159
216 ssd 0.87329 0.87329 osd.216
-28 280.04810 280.04810 host NODE-9
178 hdd 14.60149 14.60149 osd.178
179 ssd 0.87329 0.87329 osd.179
201 ssd 0.87329 0.87329 osd.201
217 ssd 0.87329 0.87329 osd.217
-31 280.04803 280.04803 host NODE-10
180 hdd 14.60149 14.60149 osd.180
199 ssd 0.87320 0.87320 osd.199
202 ssd 0.87329 0.87329 osd.202
218 ssd 0.87329 0.87329 osd.218
*This pg "6.dc" is on 199,213,217 OSD's. *
6.dc 812 0 0 0 0 1369675264
0 0 3005 3005 active+clean 2021-08-31
16:36:06.645208 32265'415965 32265:287175109
[199,213,217]
*ceph osd df tree | grep "CLASS\|ssd" | grep ".199\|.213\|217"*
199 ssd 0.87320 1.00000 894 GiB 281 GiB 119 GiB 159 GiB 2.5 GiB 614
GiB 31.38 0.52 103 up osd.199
213 ssd 0.87329 1.00000 894 GiB 291 GiB 95 GiB 195 GiB 2.3 GiB 603
GiB 32.59 0.54 95 up osd.213
217 ssd 0.87329 1.00000 894 GiB 261 GiB 83 GiB 176 GiB 2.3 GiB 633
GiB 29.18 0.48 89 up osd.217
As you can see the pg lives on 3 ssd OSD's and one of them is the new one.
So we can not say it belongs to someone else.
rule ssd-rule {
id 1
type replicated
step take default class ssd
step chooseleaf firstn 0 type host
step emit
}
pool 54 'rgw.buckets.index' replicated size 3 min_size 1 crush_rule 1
object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change
31607 lfor 0/0/30823 flags hashpspool stripe_width 0 compression_algorithm
lz4 compression_mode aggressive application rgw
What is the next step?
Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>, 1 Eyl 2021 Çar, 04:03 tarihinde
şunu yazdı:
> Yeah, I would suggest inspecting your CRUSH tree. Unfortunately the
> grep above removed that information from 'df tree', but from the
> information you provided there does appear to be a significant
> imbalance remaining.
>
> Josh
>
> On Tue, Aug 31, 2021 at 6:02 PM mhnx <morphinwithyou@xxxxxxxxx> wrote:
> >
> > Hello Josh!
> >
> > I use balancer active - crush-compat. Balance is done and there are no
> remapped pgs at ceph -s
> >
> > ceph osd df tree | grep 'CLASS\|ssd'
> >
> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> AVAIL %USE VAR PGS STATUS TYPE NAME
> > 19 ssd 0.87320 1.00000 894 GiB 402 GiB 117 GiB 281 GiB 3.0 GiB
> 492 GiB 44.93 0.74 102 up osd.19
> > 208 ssd 0.87329 1.00000 894 GiB 205 GiB 85 GiB 113 GiB 6.6 GiB
> 690 GiB 22.89 0.38 95 up osd.208
> > 209 ssd 0.87329 1.00000 894 GiB 204 GiB 87 GiB 114 GiB 2.7 GiB
> 690 GiB 22.84 0.38 65 up osd.209
> > 199 ssd 0.87320 1.00000 894 GiB 281 GiB 118 GiB 159 GiB 2.8 GiB
> 614 GiB 31.37 0.52 103 up osd.199
> > 202 ssd 0.87329 1.00000 894 GiB 278 GiB 89 GiB 183 GiB 6.3 GiB
> 616 GiB 31.08 0.51 97 up osd.202
> > 218 ssd 0.87329 1.00000 894 GiB 201 GiB 75 GiB 124 GiB 1.8 GiB
> 693 GiB 22.46 0.37 84 up osd.218
> > 39 ssd 0.87320 1.00000 894 GiB 334 GiB 86 GiB 242 GiB 5.3 GiB
> 560 GiB 37.34 0.61 91 up osd.39
> > 207 ssd 0.87329 1.00000 894 GiB 232 GiB 88 GiB 138 GiB 7.0 GiB
> 662 GiB 25.99 0.43 81 up osd.207
> > 210 ssd 0.87329 1.00000 894 GiB 270 GiB 109 GiB 160 GiB 1.4 GiB
> 624 GiB 30.18 0.50 99 up osd.210
> > 59 ssd 0.87320 1.00000 894 GiB 374 GiB 127 GiB 244 GiB 3.1 GiB
> 520 GiB 41.79 0.69 97 up osd.59
> > 203 ssd 0.87329 1.00000 894 GiB 314 GiB 96 GiB 210 GiB 7.5 GiB
> 581 GiB 35.06 0.58 104 up osd.203
> > 211 ssd 0.87329 1.00000 894 GiB 231 GiB 60 GiB 169 GiB 1.7 GiB
> 663 GiB 25.82 0.42 81 up osd.211
> > 79 ssd 0.87320 1.00000 894 GiB 409 GiB 109 GiB 298 GiB 2.0 GiB
> 486 GiB 45.70 0.75 102 up osd.79
> > 206 ssd 0.87329 1.00000 894 GiB 284 GiB 107 GiB 175 GiB 1.9 GiB
> 610 GiB 31.79 0.52 94 up osd.206
> > 212 ssd 0.87329 1.00000 894 GiB 239 GiB 85 GiB 152 GiB 2.0 GiB
> 655 GiB 26.71 0.44 80 up osd.212
> > 99 ssd 0.87320 1.00000 894 GiB 392 GiB 73 GiB 314 GiB 4.7 GiB
> 503 GiB 43.79 0.72 85 up osd.99
> > 205 ssd 0.87329 1.00000 894 GiB 445 GiB 87 GiB 353 GiB 4.8 GiB
> 449 GiB 49.80 0.82 95 up osd.205
> > 213 ssd 0.87329 1.00000 894 GiB 291 GiB 94 GiB 194 GiB 2.3 GiB
> 603 GiB 32.57 0.54 95 up osd.213
> > 114 ssd 0.87329 1.00000 894 GiB 319 GiB 125 GiB 191 GiB 3.0 GiB
> 575 GiB 35.67 0.59 99 up osd.114
> > 200 ssd 0.87329 1.00000 894 GiB 231 GiB 78 GiB 150 GiB 2.9 GiB
> 663 GiB 25.83 0.42 90 up osd.200
> > 214 ssd 0.87329 1.00000 894 GiB 296 GiB 106 GiB 187 GiB 2.6 GiB
> 598 GiB 33.09 0.54 100 up osd.214
> > 139 ssd 0.87320 1.00000 894 GiB 270 GiB 98 GiB 169 GiB 2.3 GiB
> 624 GiB 30.18 0.50 96 up osd.139
> > 204 ssd 0.87329 1.00000 894 GiB 301 GiB 117 GiB 181 GiB 2.9 GiB
> 593 GiB 33.64 0.55 104 up osd.204
> > 215 ssd 0.87329 1.00000 894 GiB 203 GiB 78 GiB 122 GiB 3.3 GiB
> 691 GiB 22.69 0.37 81 up osd.215
> > 119 ssd 0.87329 1.00000 894 GiB 200 GiB 106 GiB 92 GiB 2.0 GiB
> 694 GiB 22.39 0.37 99 up osd.119
> > 159 ssd 0.87329 1.00000 894 GiB 213 GiB 96 GiB 113 GiB 3.2 GiB
> 682 GiB 23.77 0.39 93 up osd.159
> > 216 ssd 0.87329 1.00000 894 GiB 322 GiB 109 GiB 211 GiB 1.8 GiB
> 573 GiB 35.96 0.59 101 up osd.216
> > 179 ssd 0.87329 1.00000 894 GiB 389 GiB 85 GiB 300 GiB 3.2 GiB
> 505 GiB 43.49 0.71 104 up osd.179
> > 201 ssd 0.87329 1.00000 894 GiB 494 GiB 104 GiB 386 GiB 4.1 GiB
> 401 GiB 55.20 0.91 103 up osd.201
> > 217 ssd 0.87329 1.00000 894 GiB 261 GiB 83 GiB 176 GiB 2.3 GiB
> 634 GiB 29.15 0.48 89 up osd.217
> >
> >
> > When I check the balancer status I saw that: ""optimize_result": "Some
> osds belong to multiple subtrees:"
> > Do I need to check crushmap?
> >
> >
> >
> > Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>, 31 Ağu 2021 Sal, 22:32
> tarihinde şunu yazdı:
> >>
> >> Hi there,
> >>
> >> Could you post the output of "ceph osd df tree"? I would highly
> >> suspect that this is a result of imbalance, and that's the easiest way
> >> to see if that's the case. It would also confirm that the new disks
> >> have taken on PGs.
> >>
> >> Josh
> >>
> >> On Tue, Aug 31, 2021 at 10:50 AM mhnx <morphinwithyou@xxxxxxxxx> wrote:
> >> >
> >> > I'm using Nautilus 14.2.16
> >> >
> >> > I was have 20 ssd OSD in my cluster and I added 10 more. " Each
> SSD=960GB"
> >> > The Size increased to *(26TiB)* as expected but the Replicated (3)
> Pool Max
> >> > Avail didn't changed *(3.5TiB)*.
> >> > I've increased pg_num and PG rebalance is also done.
> >> >
> >> > Do I need any special treatment to expand the pool Max Avail?
> >> >
> >> > CLASS SIZE AVAIL USED RAW USED %RAW USED
> >> > hdd 2.7 PiB 1.0 PiB 1.6 PiB 1.6 PiB
> 61.12
> >> > ssd *26 TiB* 18 TiB 2.8 TiB 8.7 TiB
> 33.11
> >> > TOTAL 2.7 PiB 1.1 PiB 1.6 PiB 1.7 PiB
> 60.85
> >> >
> >> > POOLS:
> >> > POOL ID PGS STORED OBJECTS
> >> > USED %USED MAX AVAIL
> >> > xxx.rgw.buckets.index 54 128 541 GiB 435.69k
> 541
> >> > GiB 4.82 *3.5 TiB*
> >> > _______________________________________________
> >> > ceph-users mailing list -- ceph-users@xxxxxxx
> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
