thx Paul - I suspect these shadow trees causing this misbehaviour.
I have a second luminous cluster where these balancer settings work as expected - this working one has hdd+ssd osds
i cannot use the upmap balancer because of some jewel-krbd clients - at least they are being reported as jewel clients
"client": {
"group": {
"features": "0x7010fb86aa42ada",
"release": "jewel",
"num": 1
},
"group": {
"features": "0x27018fb86aa42ada",
"release": "jewel",
"num": 3
},
is there a good way to decode the "features" value?
wolfgang
Am 26.06.2019 um 13:29 schrieb Paul
Emmerich:
Device classes are implemented with magic invisible crush
trees; you've got two completely independent trees internally:
one for crush rules mapping to HDDs, one to legacy crush rules
not specifying a device class.
The balancer *should* be aware of this and ignore it, but
I'm not sure about the state of the balancer on Luminous.
There were quite a few problems in older versions, lots of
them have been fixed in backports.
The upmap balancer is much better than the crush-compat
balancer, but it requires all clients to run Luminous or
later.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
Hi,
tried to enable the ceph balancer on a 12.2.12 cluster and got this:
mgr[balancer] Some osds belong to multiple subtrees: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305]
I'm not aware of any additional subtree - maybe someone can enlighten me:
ceph balancer status
{
"active": true,
"plans": [],
"mode": "crush-compat"
}
ceph osd crush tree
ID CLASS WEIGHT (compat) TYPE NAME
-1 3176.04785 root default
-7 316.52490 316.52490 host node0
0 hdd 9.09560 9.09560 osd.0
4 hdd 9.09560 9.09560 osd.4
8 hdd 9.09560 9.09560 osd.8
10 hdd 9.09560 9.09560 osd.10
12 hdd 9.09560 9.09560 osd.12
16 hdd 9.09560 9.09560 osd.16
20 hdd 9.09560 9.09560 osd.20
21 hdd 9.09560 9.09560 osd.21
26 hdd 9.09560 9.09560 osd.26
29 hdd 9.09560 9.09560 osd.29
31 hdd 9.09560 9.09560 osd.31
35 hdd 9.09560 9.09560 osd.35
37 hdd 9.09560 9.09560 osd.37
44 hdd 9.09560 9.09560 osd.44
47 hdd 9.09560 9.09560 osd.47
56 hdd 9.09560 9.09560 osd.56
59 hdd 9.09560 9.09560 osd.59
65 hdd 9.09560 9.09560 osd.65
71 hdd 9.09560 9.09560 osd.71
77 hdd 9.09560 9.09560 osd.77
80 hdd 9.09560 9.09560 osd.80
83 hdd 9.09569 9.09569 osd.83
86 hdd 9.09560 9.09560 osd.86
88 hdd 9.09560 9.09560 osd.88
94 hdd 10.91409 10.91409 osd.94
95 hdd 10.91409 10.91409 osd.95
98 hdd 10.91409 10.91409 osd.98
99 hdd 10.91409 10.91409 osd.99
238 hdd 9.09569 9.09569 osd.238
239 hdd 9.09569 9.09569 osd.239
240 hdd 9.09569 9.09569 osd.240
241 hdd 9.09569 9.09569 osd.241
242 hdd 9.09569 9.09569 osd.242
243 hdd 9.09569 9.09569 osd.243
-3 316.52426 316.52426 host node1
1 hdd 9.09560 9.09560 osd.1
5 hdd 9.09560 9.09560 osd.5
6 hdd 9.09560 9.09560 osd.6
11 hdd 9.09560 9.09560 osd.11
13 hdd 9.09560 9.09560 osd.13
15 hdd 9.09560 9.09560 osd.15
19 hdd 9.09560 9.09560 osd.19
23 hdd 9.09560 9.09560 osd.23
25 hdd 9.09560 9.09560 osd.25
28 hdd 9.09560 9.09560 osd.28
32 hdd 9.09560 9.09560 osd.32
34 hdd 9.09560 9.09560 osd.34
38 hdd 9.09560 9.09560 osd.38
41 hdd 9.09560 9.09560 osd.41
43 hdd 9.09560 9.09560 osd.43
46 hdd 9.09560 9.09560 osd.46
49 hdd 9.09560 9.09560 osd.49
52 hdd 9.09560 9.09560 osd.52
55 hdd 9.09560 9.09560 osd.55
58 hdd 9.09560 9.09560 osd.58
61 hdd 9.09560 9.09560 osd.61
64 hdd 9.09560 9.09560 osd.64
67 hdd 9.09560 9.09560 osd.67
70 hdd 9.09560 9.09560 osd.70
73 hdd 9.09560 9.09560 osd.73
76 hdd 9.09560 9.09560 osd.76
79 hdd 9.09560 9.09560 osd.79
81 hdd 9.09560 9.09560 osd.81
85 hdd 9.09560 9.09560 osd.85
89 hdd 9.09560 9.09560 osd.89
90 hdd 10.91409 10.91409 osd.90
91 hdd 10.91409 10.91409 osd.91
96 hdd 10.91409 10.91409 osd.96
97 hdd 10.91409 10.91409 osd.97
-5 316.52426 316.52426 host node2
2 hdd 9.09560 9.09560 osd.2
3 hdd 9.09560 9.09560 osd.3
7 hdd 9.09560 9.09560 osd.7
9 hdd 9.09560 9.09560 osd.9
14 hdd 9.09560 9.09560 osd.14
17 hdd 9.09560 9.09560 osd.17
18 hdd 9.09560 9.09560 osd.18
22 hdd 9.09560 9.09560 osd.22
24 hdd 9.09560 9.09560 osd.24
27 hdd 9.09560 9.09560 osd.27
30 hdd 9.09560 9.09560 osd.30
33 hdd 9.09560 9.09560 osd.33
36 hdd 9.09560 9.09560 osd.36
39 hdd 9.09560 9.09560 osd.39
42 hdd 9.09560 9.09560 osd.42
45 hdd 9.09560 9.09560 osd.45
48 hdd 9.09560 9.09560 osd.48
51 hdd 9.09560 9.09560 osd.51
54 hdd 9.09560 9.09560 osd.54
57 hdd 9.09560 9.09560 osd.57
60 hdd 9.09560 9.09560 osd.60
63 hdd 9.09560 9.09560 osd.63
66 hdd 9.09560 9.09560 osd.66
69 hdd 9.09560 9.09560 osd.69
72 hdd 9.09560 9.09560 osd.72
75 hdd 9.09560 9.09560 osd.75
78 hdd 9.09560 9.09560 osd.78
82 hdd 9.09560 9.09560 osd.82
84 hdd 9.09560 9.09560 osd.84
87 hdd 9.09560 9.09560 osd.87
92 hdd 10.91409 10.91409 osd.92
93 hdd 10.91409 10.91409 osd.93
100 hdd 10.91409 10.91409 osd.100
101 hdd 10.91409 10.91409 osd.101
-13 371.07864 371.07864 host node3
102 hdd 10.91408 10.91408 osd.102
103 hdd 10.91408 10.91408 osd.103
104 hdd 10.91408 10.91408 osd.104
105 hdd 10.91408 10.91408 osd.105
106 hdd 10.91408 10.91408 osd.106
107 hdd 10.91408 10.91408 osd.107
108 hdd 10.91408 10.91408 osd.108
109 hdd 10.91408 10.91408 osd.109
110 hdd 10.91408 10.91408 osd.110
111 hdd 10.91408 10.91408 osd.111
112 hdd 10.91408 10.91408 osd.112
113 hdd 10.91408 10.91408 osd.113
114 hdd 10.91408 10.91408 osd.114
115 hdd 10.91408 10.91408 osd.115
116 hdd 10.91408 10.91408 osd.116
117 hdd 10.91408 10.91408 osd.117
118 hdd 10.91408 10.91408 osd.118
119 hdd 10.91408 10.91408 osd.119
120 hdd 10.91408 10.91408 osd.120
121 hdd 10.91408 10.91408 osd.121
122 hdd 10.91408 10.91408 osd.122
123 hdd 10.91408 10.91408 osd.123
124 hdd 10.91408 10.91408 osd.124
125 hdd 10.91408 10.91408 osd.125
126 hdd 10.91408 10.91408 osd.126
127 hdd 10.91408 10.91408 osd.127
128 hdd 10.91408 10.91408 osd.128
129 hdd 10.91408 10.91408 osd.129
130 hdd 10.91408 10.91408 osd.130
131 hdd 10.91408 10.91408 osd.131
132 hdd 10.91408 10.91408 osd.132
133 hdd 10.91408 10.91408 osd.133
134 hdd 10.91408 10.91408 osd.134
135 hdd 10.91408 10.91408 osd.135
-16 371.07916 371.07916 host node4
136 hdd 10.91409 10.91409 osd.136
137 hdd 10.91409 10.91409 osd.137
138 hdd 10.91409 10.91409 osd.138
139 hdd 10.91409 10.91409 osd.139
140 hdd 10.91409 10.91409 osd.140
141 hdd 10.91409 10.91409 osd.141
142 hdd 10.91409 10.91409 osd.142
143 hdd 10.91409 10.91409 osd.143
144 hdd 10.91409 10.91409 osd.144
145 hdd 10.91409 10.91409 osd.145
146 hdd 10.91409 10.91409 osd.146
147 hdd 10.91409 10.91409 osd.147
148 hdd 10.91409 10.91409 osd.148
149 hdd 10.91409 10.91409 osd.149
150 hdd 10.91409 10.91409 osd.150
151 hdd 10.91409 10.91409 osd.151
152 hdd 10.91409 10.91409 osd.152
153 hdd 10.91409 10.91409 osd.153
154 hdd 10.91409 10.91409 osd.154
155 hdd 10.91409 10.91409 osd.155
156 hdd 10.91409 10.91409 osd.156
157 hdd 10.91409 10.91409 osd.157
158 hdd 10.91409 10.91409 osd.158
159 hdd 10.91409 10.91409 osd.159
160 hdd 10.91409 10.91409 osd.160
161 hdd 10.91409 10.91409 osd.161
162 hdd 10.91409 10.91409 osd.162
163 hdd 10.91409 10.91409 osd.163
164 hdd 10.91409 10.91409 osd.164
165 hdd 10.91409 10.91409 osd.165
166 hdd 10.91409 10.91409 osd.166
167 hdd 10.91409 10.91409 osd.167
168 hdd 10.91409 10.91409 osd.168
169 hdd 10.91409 10.91409 osd.169
-19 371.07916 371.07916 host node5
170 hdd 10.91409 10.91409 osd.170
171 hdd 10.91409 10.91409 osd.171
173 hdd 10.91409 10.91409 osd.173
174 hdd 10.91409 10.91409 osd.174
175 hdd 10.91409 10.91409 osd.175
176 hdd 10.91409 10.91409 osd.176
177 hdd 10.91409 10.91409 osd.177
178 hdd 10.91409 10.91409 osd.178
179 hdd 10.91409 10.91409 osd.179
180 hdd 10.91409 10.91409 osd.180
181 hdd 10.91409 10.91409 osd.181
182 hdd 10.91409 10.91409 osd.182
183 hdd 10.91409 10.91409 osd.183
184 hdd 10.91409 10.91409 osd.184
185 hdd 10.91409 10.91409 osd.185
186 hdd 10.91409 10.91409 osd.186
187 hdd 10.91409 10.91409 osd.187
188 hdd 10.91409 10.91409 osd.188
189 hdd 10.91409 10.91409 osd.189
190 hdd 10.91409 10.91409 osd.190
191 hdd 10.91409 10.91409 osd.191
192 hdd 10.91409 10.91409 osd.192
193 hdd 10.91409 10.91409 osd.193
194 hdd 10.91409 10.91409 osd.194
195 hdd 10.91409 10.91409 osd.195
196 hdd 10.91409 10.91409 osd.196
197 hdd 10.91409 10.91409 osd.197
198 hdd 10.91409 10.91409 osd.198
199 hdd 10.91409 10.91409 osd.199
200 hdd 10.91409 10.91409 osd.200
201 hdd 10.91409 10.91409 osd.201
202 hdd 10.91409 10.91409 osd.202
203 hdd 10.91409 10.91409 osd.203
237 hdd 10.91409 10.91409 osd.237
-22 371.07916 371.07916 host node6
172 hdd 10.91409 10.91409 osd.172
204 hdd 10.91409 10.91409 osd.204
205 hdd 10.91409 10.91409 osd.205
206 hdd 10.91409 10.91409 osd.206
207 hdd 10.91409 10.91409 osd.207
208 hdd 10.91409 10.91409 osd.208
209 hdd 10.91409 10.91409 osd.209
210 hdd 10.91409 10.91409 osd.210
211 hdd 10.91409 10.91409 osd.211
212 hdd 10.91409 10.91409 osd.212
213 hdd 10.91409 10.91409 osd.213
214 hdd 10.91409 10.91409 osd.214
215 hdd 10.91409 10.91409 osd.215
216 hdd 10.91409 10.91409 osd.216
217 hdd 10.91409 10.91409 osd.217
218 hdd 10.91409 10.91409 osd.218
219 hdd 10.91409 10.91409 osd.219
220 hdd 10.91409 10.91409 osd.220
221 hdd 10.91409 10.91409 osd.221
222 hdd 10.91409 10.91409 osd.222
223 hdd 10.91409 10.91409 osd.223
224 hdd 10.91409 10.91409 osd.224
225 hdd 10.91409 10.91409 osd.225
226 hdd 10.91409 10.91409 osd.226
227 hdd 10.91409 10.91409 osd.227
228 hdd 10.91409 10.91409 osd.228
229 hdd 10.91409 10.91409 osd.229
230 hdd 10.91409 10.91409 osd.230
231 hdd 10.91409 10.91409 osd.231
232 hdd 10.91409 10.91409 osd.232
233 hdd 10.91409 10.91409 osd.233
234 hdd 10.91409 10.91409 osd.234
235 hdd 10.91409 10.91409 osd.235
236 hdd 10.91409 10.91409 osd.236
-28 371.07916 371.07916 host node7
50 hdd 10.91409 10.91409 osd.50
53 hdd 10.91409 10.91409 osd.53
62 hdd 10.91409 10.91409 osd.62
68 hdd 10.91409 10.91409 osd.68
74 hdd 10.91409 10.91409 osd.74
244 hdd 10.91409 10.91409 osd.244
245 hdd 10.91409 10.91409 osd.245
246 hdd 10.91409 10.91409 osd.246
247 hdd 10.91409 10.91409 osd.247
248 hdd 10.91409 10.91409 osd.248
249 hdd 10.91409 10.91409 osd.249
250 hdd 10.91409 10.91409 osd.250
251 hdd 10.91409 10.91409 osd.251
252 hdd 10.91409 10.91409 osd.252
253 hdd 10.91409 10.91409 osd.253
254 hdd 10.91409 10.91409 osd.254
255 hdd 10.91409 10.91409 osd.255
256 hdd 10.91409 10.91409 osd.256
257 hdd 10.91409 10.91409 osd.257
258 hdd 10.91409 10.91409 osd.258
259 hdd 10.91409 10.91409 osd.259
260 hdd 10.91409 10.91409 osd.260
261 hdd 10.91409 10.91409 osd.261
262 hdd 10.91409 10.91409 osd.262
263 hdd 10.91409 10.91409 osd.263
264 hdd 10.91409 10.91409 osd.264
265 hdd 10.91409 10.91409 osd.265
266 hdd 10.91409 10.91409 osd.266
267 hdd 10.91409 10.91409 osd.267
268 hdd 10.91409 10.91409 osd.268
269 hdd 10.91409 10.91409 osd.269
270 hdd 10.91409 10.91409 osd.270
271 hdd 10.91409 10.91409 osd.271
272 hdd 10.91409 10.91409 osd.272
-25 371.07916 371.07916 host node8
40 hdd 10.91409 10.91409 osd.40
273 hdd 10.91409 10.91409 osd.273
274 hdd 10.91409 10.91409 osd.274
275 hdd 10.91409 10.91409 osd.275
276 hdd 10.91409 10.91409 osd.276
277 hdd 10.91409 10.91409 osd.277
278 hdd 10.91409 10.91409 osd.278
279 hdd 10.91409 10.91409 osd.279
280 hdd 10.91409 10.91409 osd.280
281 hdd 10.91409 10.91409 osd.281
282 hdd 10.91409 10.91409 osd.282
283 hdd 10.91409 10.91409 osd.283
284 hdd 10.91409 10.91409 osd.284
285 hdd 10.91409 10.91409 osd.285
286 hdd 10.91409 10.91409 osd.286
287 hdd 10.91409 10.91409 osd.287
288 hdd 10.91409 10.91409 osd.288
289 hdd 10.91409 10.91409 osd.289
290 hdd 10.91409 10.91409 osd.290
291 hdd 10.91409 10.91409 osd.291
292 hdd 10.91409 10.91409 osd.292
293 hdd 10.91409 10.91409 osd.293
294 hdd 10.91409 10.91409 osd.294
295 hdd 10.91409 10.91409 osd.295
296 hdd 10.91409 10.91409 osd.296
297 hdd 10.91409 10.91409 osd.297
298 hdd 10.91409 10.91409 osd.298
299 hdd 10.91409 10.91409 osd.299
300 hdd 10.91409 10.91409 osd.300
301 hdd 10.91409 10.91409 osd.301
302 hdd 10.91409 10.91409 osd.302
303 hdd 10.91409 10.91409 osd.303
304 hdd 10.91409 10.91409 osd.304
305 hdd 10.91409 10.91409 osd.305
bye
wolfgang
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
|
