Device classes are implemented with magic invisible crush trees; you've got two completely independent trees internally: one for crush rules mapping to HDDs, one to legacy crush rules not specifying a device class.
The balancer *should* be aware of this and ignore it, but I'm not sure about the state of the balancer on Luminous. There were quite a few problems in older versions, lots of them have been fixed in backports.
The upmap balancer is much better than the crush-compat balancer, but it requires all clients to run Luminous or later.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Wed, Jun 26, 2019 at 10:21 AM Wolfgang Lendl <wolfgang.lendl@xxxxxxxxxxxxxxxx> wrote:
_______________________________________________Hi,tried to enable the ceph balancer on a 12.2.12 cluster and got this: mgr[balancer] Some osds belong to multiple subtrees: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305] I'm not aware of any additional subtree - maybe someone can enlighten me: ceph balancer status { "active": true, "plans": [], "mode": "crush-compat" } ceph osd crush tree ID CLASS WEIGHT (compat) TYPE NAME -1 3176.04785 root default -7 316.52490 316.52490 host node0 0 hdd 9.09560 9.09560 osd.0 4 hdd 9.09560 9.09560 osd.4 8 hdd 9.09560 9.09560 osd.8 10 hdd 9.09560 9.09560 osd.10 12 hdd 9.09560 9.09560 osd.12 16 hdd 9.09560 9.09560 osd.16 20 hdd 9.09560 9.09560 osd.20 21 hdd 9.09560 9.09560 osd.21 26 hdd 9.09560 9.09560 osd.26 29 hdd 9.09560 9.09560 osd.29 31 hdd 9.09560 9.09560 osd.31 35 hdd 9.09560 9.09560 osd.35 37 hdd 9.09560 9.09560 osd.37 44 hdd 9.09560 9.09560 osd.44 47 hdd 9.09560 9.09560 osd.47 56 hdd 9.09560 9.09560 osd.56 59 hdd 9.09560 9.09560 osd.59 65 hdd 9.09560 9.09560 osd.65 71 hdd 9.09560 9.09560 osd.71 77 hdd 9.09560 9.09560 osd.77 80 hdd 9.09560 9.09560 osd.80 83 hdd 9.09569 9.09569 osd.83 86 hdd 9.09560 9.09560 osd.86 88 hdd 9.09560 9.09560 osd.88 94 hdd 10.91409 10.91409 osd.94 95 hdd 10.91409 10.91409 osd.95 98 hdd 10.91409 10.91409 osd.98 99 hdd 10.91409 10.91409 osd.99 238 hdd 9.09569 9.09569 osd.238 239 hdd 9.09569 9.09569 osd.239 240 hdd 9.09569 9.09569 osd.240 241 hdd 9.09569 9.09569 osd.241 242 hdd 9.09569 9.09569 osd.242 243 hdd 9.09569 9.09569 osd.243 -3 316.52426 316.52426 host node1 1 hdd 9.09560 9.09560 osd.1 5 hdd 9.09560 9.09560 osd.5 6 hdd 9.09560 9.09560 osd.6 11 hdd 9.09560 9.09560 osd.11 13 hdd 9.09560 9.09560 osd.13 15 hdd 9.09560 9.09560 osd.15 19 hdd 9.09560 9.09560 osd.19 23 hdd 9.09560 9.09560 osd.23 25 hdd 9.09560 9.09560 osd.25 28 hdd 9.09560 9.09560 osd.28 32 hdd 9.09560 9.09560 osd.32 34 hdd 9.09560 9.09560 osd.34 38 hdd 9.09560 9.09560 osd.38 41 hdd 9.09560 9.09560 osd.41 43 hdd 9.09560 9.09560 osd.43 46 hdd 9.09560 9.09560 osd.46 49 hdd 9.09560 9.09560 osd.49 52 hdd 9.09560 9.09560 osd.52 55 hdd 9.09560 9.09560 osd.55 58 hdd 9.09560 9.09560 osd.58 61 hdd 9.09560 9.09560 osd.61 64 hdd 9.09560 9.09560 osd.64 67 hdd 9.09560 9.09560 osd.67 70 hdd 9.09560 9.09560 osd.70 73 hdd 9.09560 9.09560 osd.73 76 hdd 9.09560 9.09560 osd.76 79 hdd 9.09560 9.09560 osd.79 81 hdd 9.09560 9.09560 osd.81 85 hdd 9.09560 9.09560 osd.85 89 hdd 9.09560 9.09560 osd.89 90 hdd 10.91409 10.91409 osd.90 91 hdd 10.91409 10.91409 osd.91 96 hdd 10.91409 10.91409 osd.96 97 hdd 10.91409 10.91409 osd.97 -5 316.52426 316.52426 host node2 2 hdd 9.09560 9.09560 osd.2 3 hdd 9.09560 9.09560 osd.3 7 hdd 9.09560 9.09560 osd.7 9 hdd 9.09560 9.09560 osd.9 14 hdd 9.09560 9.09560 osd.14 17 hdd 9.09560 9.09560 osd.17 18 hdd 9.09560 9.09560 osd.18 22 hdd 9.09560 9.09560 osd.22 24 hdd 9.09560 9.09560 osd.24 27 hdd 9.09560 9.09560 osd.27 30 hdd 9.09560 9.09560 osd.30 33 hdd 9.09560 9.09560 osd.33 36 hdd 9.09560 9.09560 osd.36 39 hdd 9.09560 9.09560 osd.39 42 hdd 9.09560 9.09560 osd.42 45 hdd 9.09560 9.09560 osd.45 48 hdd 9.09560 9.09560 osd.48 51 hdd 9.09560 9.09560 osd.51 54 hdd 9.09560 9.09560 osd.54 57 hdd 9.09560 9.09560 osd.57 60 hdd 9.09560 9.09560 osd.60 63 hdd 9.09560 9.09560 osd.63 66 hdd 9.09560 9.09560 osd.66 69 hdd 9.09560 9.09560 osd.69 72 hdd 9.09560 9.09560 osd.72 75 hdd 9.09560 9.09560 osd.75 78 hdd 9.09560 9.09560 osd.78 82 hdd 9.09560 9.09560 osd.82 84 hdd 9.09560 9.09560 osd.84 87 hdd 9.09560 9.09560 osd.87 92 hdd 10.91409 10.91409 osd.92 93 hdd 10.91409 10.91409 osd.93 100 hdd 10.91409 10.91409 osd.100 101 hdd 10.91409 10.91409 osd.101 -13 371.07864 371.07864 host node3 102 hdd 10.91408 10.91408 osd.102 103 hdd 10.91408 10.91408 osd.103 104 hdd 10.91408 10.91408 osd.104 105 hdd 10.91408 10.91408 osd.105 106 hdd 10.91408 10.91408 osd.106 107 hdd 10.91408 10.91408 osd.107 108 hdd 10.91408 10.91408 osd.108 109 hdd 10.91408 10.91408 osd.109 110 hdd 10.91408 10.91408 osd.110 111 hdd 10.91408 10.91408 osd.111 112 hdd 10.91408 10.91408 osd.112 113 hdd 10.91408 10.91408 osd.113 114 hdd 10.91408 10.91408 osd.114 115 hdd 10.91408 10.91408 osd.115 116 hdd 10.91408 10.91408 osd.116 117 hdd 10.91408 10.91408 osd.117 118 hdd 10.91408 10.91408 osd.118 119 hdd 10.91408 10.91408 osd.119 120 hdd 10.91408 10.91408 osd.120 121 hdd 10.91408 10.91408 osd.121 122 hdd 10.91408 10.91408 osd.122 123 hdd 10.91408 10.91408 osd.123 124 hdd 10.91408 10.91408 osd.124 125 hdd 10.91408 10.91408 osd.125 126 hdd 10.91408 10.91408 osd.126 127 hdd 10.91408 10.91408 osd.127 128 hdd 10.91408 10.91408 osd.128 129 hdd 10.91408 10.91408 osd.129 130 hdd 10.91408 10.91408 osd.130 131 hdd 10.91408 10.91408 osd.131 132 hdd 10.91408 10.91408 osd.132 133 hdd 10.91408 10.91408 osd.133 134 hdd 10.91408 10.91408 osd.134 135 hdd 10.91408 10.91408 osd.135 -16 371.07916 371.07916 host node4 136 hdd 10.91409 10.91409 osd.136 137 hdd 10.91409 10.91409 osd.137 138 hdd 10.91409 10.91409 osd.138 139 hdd 10.91409 10.91409 osd.139 140 hdd 10.91409 10.91409 osd.140 141 hdd 10.91409 10.91409 osd.141 142 hdd 10.91409 10.91409 osd.142 143 hdd 10.91409 10.91409 osd.143 144 hdd 10.91409 10.91409 osd.144 145 hdd 10.91409 10.91409 osd.145 146 hdd 10.91409 10.91409 osd.146 147 hdd 10.91409 10.91409 osd.147 148 hdd 10.91409 10.91409 osd.148 149 hdd 10.91409 10.91409 osd.149 150 hdd 10.91409 10.91409 osd.150 151 hdd 10.91409 10.91409 osd.151 152 hdd 10.91409 10.91409 osd.152 153 hdd 10.91409 10.91409 osd.153 154 hdd 10.91409 10.91409 osd.154 155 hdd 10.91409 10.91409 osd.155 156 hdd 10.91409 10.91409 osd.156 157 hdd 10.91409 10.91409 osd.157 158 hdd 10.91409 10.91409 osd.158 159 hdd 10.91409 10.91409 osd.159 160 hdd 10.91409 10.91409 osd.160 161 hdd 10.91409 10.91409 osd.161 162 hdd 10.91409 10.91409 osd.162 163 hdd 10.91409 10.91409 osd.163 164 hdd 10.91409 10.91409 osd.164 165 hdd 10.91409 10.91409 osd.165 166 hdd 10.91409 10.91409 osd.166 167 hdd 10.91409 10.91409 osd.167 168 hdd 10.91409 10.91409 osd.168 169 hdd 10.91409 10.91409 osd.169 -19 371.07916 371.07916 host node5 170 hdd 10.91409 10.91409 osd.170 171 hdd 10.91409 10.91409 osd.171 173 hdd 10.91409 10.91409 osd.173 174 hdd 10.91409 10.91409 osd.174 175 hdd 10.91409 10.91409 osd.175 176 hdd 10.91409 10.91409 osd.176 177 hdd 10.91409 10.91409 osd.177 178 hdd 10.91409 10.91409 osd.178 179 hdd 10.91409 10.91409 osd.179 180 hdd 10.91409 10.91409 osd.180 181 hdd 10.91409 10.91409 osd.181 182 hdd 10.91409 10.91409 osd.182 183 hdd 10.91409 10.91409 osd.183 184 hdd 10.91409 10.91409 osd.184 185 hdd 10.91409 10.91409 osd.185 186 hdd 10.91409 10.91409 osd.186 187 hdd 10.91409 10.91409 osd.187 188 hdd 10.91409 10.91409 osd.188 189 hdd 10.91409 10.91409 osd.189 190 hdd 10.91409 10.91409 osd.190 191 hdd 10.91409 10.91409 osd.191 192 hdd 10.91409 10.91409 osd.192 193 hdd 10.91409 10.91409 osd.193 194 hdd 10.91409 10.91409 osd.194 195 hdd 10.91409 10.91409 osd.195 196 hdd 10.91409 10.91409 osd.196 197 hdd 10.91409 10.91409 osd.197 198 hdd 10.91409 10.91409 osd.198 199 hdd 10.91409 10.91409 osd.199 200 hdd 10.91409 10.91409 osd.200 201 hdd 10.91409 10.91409 osd.201 202 hdd 10.91409 10.91409 osd.202 203 hdd 10.91409 10.91409 osd.203 237 hdd 10.91409 10.91409 osd.237 -22 371.07916 371.07916 host node6 172 hdd 10.91409 10.91409 osd.172 204 hdd 10.91409 10.91409 osd.204 205 hdd 10.91409 10.91409 osd.205 206 hdd 10.91409 10.91409 osd.206 207 hdd 10.91409 10.91409 osd.207 208 hdd 10.91409 10.91409 osd.208 209 hdd 10.91409 10.91409 osd.209 210 hdd 10.91409 10.91409 osd.210 211 hdd 10.91409 10.91409 osd.211 212 hdd 10.91409 10.91409 osd.212 213 hdd 10.91409 10.91409 osd.213 214 hdd 10.91409 10.91409 osd.214 215 hdd 10.91409 10.91409 osd.215 216 hdd 10.91409 10.91409 osd.216 217 hdd 10.91409 10.91409 osd.217 218 hdd 10.91409 10.91409 osd.218 219 hdd 10.91409 10.91409 osd.219 220 hdd 10.91409 10.91409 osd.220 221 hdd 10.91409 10.91409 osd.221 222 hdd 10.91409 10.91409 osd.222 223 hdd 10.91409 10.91409 osd.223 224 hdd 10.91409 10.91409 osd.224 225 hdd 10.91409 10.91409 osd.225 226 hdd 10.91409 10.91409 osd.226 227 hdd 10.91409 10.91409 osd.227 228 hdd 10.91409 10.91409 osd.228 229 hdd 10.91409 10.91409 osd.229 230 hdd 10.91409 10.91409 osd.230 231 hdd 10.91409 10.91409 osd.231 232 hdd 10.91409 10.91409 osd.232 233 hdd 10.91409 10.91409 osd.233 234 hdd 10.91409 10.91409 osd.234 235 hdd 10.91409 10.91409 osd.235 236 hdd 10.91409 10.91409 osd.236 -28 371.07916 371.07916 host node7 50 hdd 10.91409 10.91409 osd.50 53 hdd 10.91409 10.91409 osd.53 62 hdd 10.91409 10.91409 osd.62 68 hdd 10.91409 10.91409 osd.68 74 hdd 10.91409 10.91409 osd.74 244 hdd 10.91409 10.91409 osd.244 245 hdd 10.91409 10.91409 osd.245 246 hdd 10.91409 10.91409 osd.246 247 hdd 10.91409 10.91409 osd.247 248 hdd 10.91409 10.91409 osd.248 249 hdd 10.91409 10.91409 osd.249 250 hdd 10.91409 10.91409 osd.250 251 hdd 10.91409 10.91409 osd.251 252 hdd 10.91409 10.91409 osd.252 253 hdd 10.91409 10.91409 osd.253 254 hdd 10.91409 10.91409 osd.254 255 hdd 10.91409 10.91409 osd.255 256 hdd 10.91409 10.91409 osd.256 257 hdd 10.91409 10.91409 osd.257 258 hdd 10.91409 10.91409 osd.258 259 hdd 10.91409 10.91409 osd.259 260 hdd 10.91409 10.91409 osd.260 261 hdd 10.91409 10.91409 osd.261 262 hdd 10.91409 10.91409 osd.262 263 hdd 10.91409 10.91409 osd.263 264 hdd 10.91409 10.91409 osd.264 265 hdd 10.91409 10.91409 osd.265 266 hdd 10.91409 10.91409 osd.266 267 hdd 10.91409 10.91409 osd.267 268 hdd 10.91409 10.91409 osd.268 269 hdd 10.91409 10.91409 osd.269 270 hdd 10.91409 10.91409 osd.270 271 hdd 10.91409 10.91409 osd.271 272 hdd 10.91409 10.91409 osd.272 -25 371.07916 371.07916 host node8 40 hdd 10.91409 10.91409 osd.40 273 hdd 10.91409 10.91409 osd.273 274 hdd 10.91409 10.91409 osd.274 275 hdd 10.91409 10.91409 osd.275 276 hdd 10.91409 10.91409 osd.276 277 hdd 10.91409 10.91409 osd.277 278 hdd 10.91409 10.91409 osd.278 279 hdd 10.91409 10.91409 osd.279 280 hdd 10.91409 10.91409 osd.280 281 hdd 10.91409 10.91409 osd.281 282 hdd 10.91409 10.91409 osd.282 283 hdd 10.91409 10.91409 osd.283 284 hdd 10.91409 10.91409 osd.284 285 hdd 10.91409 10.91409 osd.285 286 hdd 10.91409 10.91409 osd.286 287 hdd 10.91409 10.91409 osd.287 288 hdd 10.91409 10.91409 osd.288 289 hdd 10.91409 10.91409 osd.289 290 hdd 10.91409 10.91409 osd.290 291 hdd 10.91409 10.91409 osd.291 292 hdd 10.91409 10.91409 osd.292 293 hdd 10.91409 10.91409 osd.293 294 hdd 10.91409 10.91409 osd.294 295 hdd 10.91409 10.91409 osd.295 296 hdd 10.91409 10.91409 osd.296 297 hdd 10.91409 10.91409 osd.297 298 hdd 10.91409 10.91409 osd.298 299 hdd 10.91409 10.91409 osd.299 300 hdd 10.91409 10.91409 osd.300 301 hdd 10.91409 10.91409 osd.301 302 hdd 10.91409 10.91409 osd.302 303 hdd 10.91409 10.91409 osd.303 304 hdd 10.91409 10.91409 osd.304 305 hdd 10.91409 10.91409 osd.305 bye wolfgang
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
