I haven't had crush-compat do anything helpful for balancing my clusters. upmap has been amazing and balanced my clusters far better than anything else I've ever seen. I would go so far as to say that upmap can achieve a perfect balance.
It seems to evenly distribute the PGs for each pool onto all OSDs that pool is on. It does that with a maximum difference of 1PG based on how divisible the number of PGs are with the number of OSDs you have. As a side note, your OSD CRUSH weights should be the default weights for their size for upmap to be as effective as it can be.
On Sat, Oct 20, 2018 at 3:58 PM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:
Ok, I'll try out the balancer end of the upcoming week then (after we've fixed a HW-issue with one of our mons
and the cooling system).
Until then, any further advice and whether upmap is recommended over crush-compat (all clients are Luminous) are welcome ;-).
Cheers,
Oliver
Am 20.10.18 um 21:26 schrieb Janne Johansson:
> Ok, can't say "why" then, I'd reweigh them somewhat to even it out,
> 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for
> the MGRs, a script or just a few manual tweaks might be in order.
>
> Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth
> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>
>> All OSDs are of the very same size. One OSD host has slightly more disks (33 instead of 31), though.
>> So also that that can't explain the hefty difference.
>>
>> I attach the output of "ceph osd tree" and "ceph osd df".
>>
>> The crush rule for the ceph_data pool is:
>> rule cephfs_data {
>> id 2
>> type erasure
>> min_size 3
>> max_size 6
>> step set_chooseleaf_tries 5
>> step set_choose_tries 100
>> step take default class hdd
>> step chooseleaf indep 0 type host
>> step emit
>> }
>> So that only considers the hdd device class. EC is done with k=4 m=2.
>>
>> So I don't see any imbalance on the hardware level, but only a somewhat uneven distribution of PGs.
>> Am I missing something, or is this really just a case for the ceph balancer plugin?
>> I'm just a bit astonished this effect is so huge.
>> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even distribution without balancing?
>> But it yields about 100 PGs per OSD, as you can see...
>>
>> --------------------------------------------------------------
>> # ceph osd tree
>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
>> -1 826.26428 root default
>> -3 0.43700 host mon001
>> 0 ssd 0.21799 osd.0 up 1.00000 1.00000
>> 1 ssd 0.21799 osd.1 up 1.00000 1.00000
>> -5 0.43700 host mon002
>> 2 ssd 0.21799 osd.2 up 1.00000 1.00000
>> 3 ssd 0.21799 osd.3 up 1.00000 1.00000
>> -31 1.81898 host mon003
>> 230 ssd 0.90999 osd.230 up 1.00000 1.00000
>> 231 ssd 0.90999 osd.231 up 1.00000 1.00000
>> -10 116.64600 host osd001
>> 4 hdd 3.64499 osd.4 up 1.00000 1.00000
>> 5 hdd 3.64499 osd.5 up 1.00000 1.00000
>> 6 hdd 3.64499 osd.6 up 1.00000 1.00000
>> 7 hdd 3.64499 osd.7 up 1.00000 1.00000
>> 8 hdd 3.64499 osd.8 up 1.00000 1.00000
>> 9 hdd 3.64499 osd.9 up 1.00000 1.00000
>> 10 hdd 3.64499 osd.10 up 1.00000 1.00000
>> 11 hdd 3.64499 osd.11 up 1.00000 1.00000
>> 12 hdd 3.64499 osd.12 up 1.00000 1.00000
>> 13 hdd 3.64499 osd.13 up 1.00000 1.00000
>> 14 hdd 3.64499 osd.14 up 1.00000 1.00000
>> 15 hdd 3.64499 osd.15 up 1.00000 1.00000
>> 16 hdd 3.64499 osd.16 up 1.00000 1.00000
>> 17 hdd 3.64499 osd.17 up 1.00000 1.00000
>> 18 hdd 3.64499 osd.18 up 1.00000 1.00000
>> 19 hdd 3.64499 osd.19 up 1.00000 1.00000
>> 20 hdd 3.64499 osd.20 up 1.00000 1.00000
>> 21 hdd 3.64499 osd.21 up 1.00000 1.00000
>> 22 hdd 3.64499 osd.22 up 1.00000 1.00000
>> 23 hdd 3.64499 osd.23 up 1.00000 1.00000
>> 24 hdd 3.64499 osd.24 up 1.00000 1.00000
>> 25 hdd 3.64499 osd.25 up 1.00000 1.00000
>> 26 hdd 3.64499 osd.26 up 1.00000 1.00000
>> 27 hdd 3.64499 osd.27 up 1.00000 1.00000
>> 28 hdd 3.64499 osd.28 up 1.00000 1.00000
>> 29 hdd 3.64499 osd.29 up 1.00000 1.00000
>> 30 hdd 3.64499 osd.30 up 1.00000 1.00000
>> 31 hdd 3.64499 osd.31 up 1.00000 1.00000
>> 32 hdd 3.64499 osd.32 up 1.00000 1.00000
>> 33 hdd 3.64499 osd.33 up 1.00000 1.00000
>> 34 hdd 3.64499 osd.34 up 1.00000 1.00000
>> 35 hdd 3.64499 osd.35 up 1.00000 1.00000
>> -13 116.64600 host osd002
>> 36 hdd 3.64499 osd.36 up 1.00000 1.00000
>> 37 hdd 3.64499 osd.37 up 1.00000 1.00000
>> 38 hdd 3.64499 osd.38 up 1.00000 1.00000
>> 39 hdd 3.64499 osd.39 up 1.00000 1.00000
>> 40 hdd 3.64499 osd.40 up 1.00000 1.00000
>> 41 hdd 3.64499 osd.41 up 1.00000 1.00000
>> 42 hdd 3.64499 osd.42 up 1.00000 1.00000
>> 43 hdd 3.64499 osd.43 up 1.00000 1.00000
>> 44 hdd 3.64499 osd.44 up 1.00000 1.00000
>> 45 hdd 3.64499 osd.45 up 1.00000 1.00000
>> 46 hdd 3.64499 osd.46 up 1.00000 1.00000
>> 47 hdd 3.64499 osd.47 up 1.00000 1.00000
>> 48 hdd 3.64499 osd.48 up 1.00000 1.00000
>> 49 hdd 3.64499 osd.49 up 1.00000 1.00000
>> 50 hdd 3.64499 osd.50 up 1.00000 1.00000
>> 51 hdd 3.64499 osd.51 up 1.00000 1.00000
>> 52 hdd 3.64499 osd.52 up 1.00000 1.00000
>> 53 hdd 3.64499 osd.53 up 1.00000 1.00000
>> 54 hdd 3.64499 osd.54 up 1.00000 1.00000
>> 55 hdd 3.64499 osd.55 up 1.00000 1.00000
>> 56 hdd 3.64499 osd.56 up 1.00000 1.00000
>> 57 hdd 3.64499 osd.57 up 1.00000 1.00000
>> 58 hdd 3.64499 osd.58 up 1.00000 1.00000
>> 59 hdd 3.64499 osd.59 up 1.00000 1.00000
>> 60 hdd 3.64499 osd.60 up 1.00000 1.00000
>> 61 hdd 3.64499 osd.61 up 1.00000 1.00000
>> 62 hdd 3.64499 osd.62 up 1.00000 1.00000
>> 63 hdd 3.64499 osd.63 up 1.00000 1.00000
>> 64 hdd 3.64499 osd.64 up 1.00000 1.00000
>> 65 hdd 3.64499 osd.65 up 1.00000 1.00000
>> 66 hdd 3.64499 osd.66 up 1.00000 1.00000
>> 67 hdd 3.64499 osd.67 up 1.00000 1.00000
>> -16 116.63899 host osd003
>> 68 hdd 3.64499 osd.68 up 1.00000 1.00000
>> 69 hdd 3.64499 osd.69 up 1.00000 1.00000
>> 70 hdd 3.64499 osd.70 up 1.00000 1.00000
>> 71 hdd 3.64499 osd.71 up 1.00000 1.00000
>> 72 hdd 3.64499 osd.72 up 1.00000 1.00000
>> 73 hdd 3.64499 osd.73 up 1.00000 1.00000
>> 74 hdd 3.64499 osd.74 up 1.00000 1.00000
>> 75 hdd 3.64499 osd.75 up 1.00000 1.00000
>> 76 hdd 3.64499 osd.76 up 1.00000 1.00000
>> 77 hdd 3.64499 osd.77 up 1.00000 1.00000
>> 78 hdd 3.64499 osd.78 up 1.00000 1.00000
>> 79 hdd 3.64499 osd.79 up 1.00000 1.00000
>> 80 hdd 3.63899 osd.80 up 1.00000 1.00000
>> 81 hdd 3.64499 osd.81 up 1.00000 1.00000
>> 82 hdd 3.64499 osd.82 up 1.00000 1.00000
>> 83 hdd 3.64499 osd.83 up 1.00000 1.00000
>> 84 hdd 3.64499 osd.84 up 1.00000 1.00000
>> 85 hdd 3.64499 osd.85 up 1.00000 1.00000
>> 86 hdd 3.64499 osd.86 up 1.00000 1.00000
>> 87 hdd 3.64499 osd.87 up 1.00000 1.00000
>> 88 hdd 3.64499 osd.88 up 1.00000 1.00000
>> 89 hdd 3.64499 osd.89 up 1.00000 1.00000
>> 90 hdd 3.64499 osd.90 up 1.00000 1.00000
>> 91 hdd 3.64499 osd.91 up 1.00000 1.00000
>> 92 hdd 3.64499 osd.92 up 1.00000 1.00000
>> 93 hdd 3.64499 osd.93 up 1.00000 1.00000
>> 94 hdd 3.64499 osd.94 up 1.00000 1.00000
>> 95 hdd 3.64499 osd.95 up 1.00000 1.00000
>> 96 hdd 3.64499 osd.96 up 1.00000 1.00000
>> 97 hdd 3.64499 osd.97 up 1.00000 1.00000
>> 98 hdd 3.64499 osd.98 up 1.00000 1.00000
>> 99 hdd 3.64499 osd.99 up 1.00000 1.00000
>> -19 116.64600 host osd004
>> 100 hdd 3.64499 osd.100 up 1.00000 1.00000
>> 101 hdd 3.64499 osd.101 up 1.00000 1.00000
>> 102 hdd 3.64499 osd.102 up 1.00000 1.00000
>> 103 hdd 3.64499 osd.103 up 1.00000 1.00000
>> 104 hdd 3.64499 osd.104 up 1.00000 1.00000
>> 105 hdd 3.64499 osd.105 up 1.00000 1.00000
>> 106 hdd 3.64499 osd.106 up 1.00000 1.00000
>> 107 hdd 3.64499 osd.107 up 1.00000 1.00000
>> 108 hdd 3.64499 osd.108 up 1.00000 1.00000
>> 109 hdd 3.64499 osd.109 up 1.00000 1.00000
>> 110 hdd 3.64499 osd.110 up 1.00000 1.00000
>> 111 hdd 3.64499 osd.111 up 1.00000 1.00000
>> 112 hdd 3.64499 osd.112 up 1.00000 1.00000
>> 113 hdd 3.64499 osd.113 up 1.00000 1.00000
>> 114 hdd 3.64499 osd.114 up 1.00000 1.00000
>> 115 hdd 3.64499 osd.115 up 1.00000 1.00000
>> 116 hdd 3.64499 osd.116 up 1.00000 1.00000
>> 117 hdd 3.64499 osd.117 up 1.00000 1.00000
>> 118 hdd 3.64499 osd.118 up 1.00000 1.00000
>> 119 hdd 3.64499 osd.119 up 1.00000 1.00000
>> 120 hdd 3.64499 osd.120 up 1.00000 1.00000
>> 121 hdd 3.64499 osd.121 up 1.00000 1.00000
>> 122 hdd 3.64499 osd.122 up 1.00000 1.00000
>> 123 hdd 3.64499 osd.123 up 1.00000 1.00000
>> 124 hdd 3.64499 osd.124 up 1.00000 1.00000
>> 125 hdd 3.64499 osd.125 up 1.00000 1.00000
>> 126 hdd 3.64499 osd.126 up 1.00000 1.00000
>> 127 hdd 3.64499 osd.127 up 1.00000 1.00000
>> 128 hdd 3.64499 osd.128 up 1.00000 1.00000
>> 129 hdd 3.64499 osd.129 up 1.00000 1.00000
>> 130 hdd 3.64499 osd.130 up 1.00000 1.00000
>> 131 hdd 3.64499 osd.131 up 1.00000 1.00000
>> -22 116.64600 host osd005
>> 132 hdd 3.64499 osd.132 up 1.00000 1.00000
>> 133 hdd 3.64499 osd.133 up 1.00000 1.00000
>> 134 hdd 3.64499 osd.134 up 1.00000 1.00000
>> 135 hdd 3.64499 osd.135 up 1.00000 1.00000
>> 136 hdd 3.64499 osd.136 up 1.00000 1.00000
>> 137 hdd 3.64499 osd.137 up 1.00000 1.00000
>> 138 hdd 3.64499 osd.138 up 1.00000 1.00000
>> 139 hdd 3.64499 osd.139 up 1.00000 1.00000
>> 140 hdd 3.64499 osd.140 up 1.00000 1.00000
>> 141 hdd 3.64499 osd.141 up 1.00000 1.00000
>> 142 hdd 3.64499 osd.142 up 1.00000 1.00000
>> 143 hdd 3.64499 osd.143 up 1.00000 1.00000
>> 144 hdd 3.64499 osd.144 up 1.00000 1.00000
>> 145 hdd 3.64499 osd.145 up 1.00000 1.00000
>> 146 hdd 3.64499 osd.146 up 1.00000 1.00000
>> 147 hdd 3.64499 osd.147 up 1.00000 1.00000
>> 148 hdd 3.64499 osd.148 up 1.00000 1.00000
>> 149 hdd 3.64499 osd.149 up 1.00000 1.00000
>> 150 hdd 3.64499 osd.150 up 1.00000 1.00000
>> 151 hdd 3.64499 osd.151 up 1.00000 1.00000
>> 152 hdd 3.64499 osd.152 up 1.00000 1.00000
>> 153 hdd 3.64499 osd.153 up 1.00000 1.00000
>> 154 hdd 3.64499 osd.154 up 1.00000 1.00000
>> 155 hdd 3.64499 osd.155 up 1.00000 1.00000
>> 156 hdd 3.64499 osd.156 up 1.00000 1.00000
>> 157 hdd 3.64499 osd.157 up 1.00000 1.00000
>> 158 hdd 3.64499 osd.158 up 1.00000 1.00000
>> 159 hdd 3.64499 osd.159 up 1.00000 1.00000
>> 160 hdd 3.64499 osd.160 up 1.00000 1.00000
>> 161 hdd 3.64499 osd.161 up 1.00000 1.00000
>> 162 hdd 3.64499 osd.162 up 1.00000 1.00000
>> 163 hdd 3.64499 osd.163 up 1.00000 1.00000
>> -25 116.63335 host osd006
>> 164 hdd 3.64499 osd.164 up 1.00000 1.00000
>> 165 hdd 3.64499 osd.165 up 1.00000 1.00000
>> 166 hdd 3.64499 osd.166 up 1.00000 1.00000
>> 167 hdd 3.64499 osd.167 up 1.00000 1.00000
>> 168 hdd 3.64499 osd.168 up 1.00000 1.00000
>> 169 hdd 3.64499 osd.169 up 1.00000 1.00000
>> 170 hdd 3.64499 osd.170 up 1.00000 1.00000
>> 171 hdd 3.64499 osd.171 up 1.00000 1.00000
>> 172 hdd 3.64499 osd.172 up 1.00000 1.00000
>> 173 hdd 3.64499 osd.173 up 1.00000 1.00000
>> 174 hdd 3.64499 osd.174 up 1.00000 1.00000
>> 175 hdd 3.64499 osd.175 up 1.00000 1.00000
>> 176 hdd 3.64499 osd.176 up 1.00000 1.00000
>> 177 hdd 3.64499 osd.177 up 1.00000 1.00000
>> 178 hdd 3.64499 osd.178 up 1.00000 1.00000
>> 179 hdd 3.64499 osd.179 up 1.00000 1.00000
>> 180 hdd 3.64499 osd.180 up 1.00000 1.00000
>> 181 hdd 3.64499 osd.181 up 1.00000 1.00000
>> 182 hdd 3.64499 osd.182 up 1.00000 1.00000
>> 183 hdd 3.64499 osd.183 up 1.00000 1.00000
>> 184 hdd 3.64499 osd.184 up 1.00000 1.00000
>> 185 hdd 3.64499 osd.185 up 1.00000 1.00000
>> 186 hdd 3.64499 osd.186 up 1.00000 1.00000
>> 187 hdd 3.64499 osd.187 up 1.00000 1.00000
>> 188 hdd 3.64499 osd.188 up 1.00000 1.00000
>> 189 hdd 3.64499 osd.189 up 1.00000 1.00000
>> 190 hdd 3.64499 osd.190 up 1.00000 1.00000
>> 191 hdd 3.64499 osd.191 up 1.00000 1.00000
>> 192 hdd 3.63869 osd.192 up 1.00000 1.00000
>> 193 hdd 3.64499 osd.193 up 1.00000 1.00000
>> 194 hdd 3.64499 osd.194 up 1.00000 1.00000
>> 195 hdd 3.64499 osd.195 up 1.00000 1.00000
>> -28 123.71500 host osd007
>> 196 hdd 3.63899 osd.196 up 1.00000 1.00000
>> 197 hdd 3.63899 osd.197 up 1.00000 1.00000
>> 198 hdd 3.63899 osd.198 up 1.00000 1.00000
>> 199 hdd 3.63899 osd.199 up 1.00000 1.00000
>> 200 hdd 3.63899 osd.200 up 1.00000 1.00000
>> 201 hdd 3.63899 osd.201 up 1.00000 1.00000
>> 202 hdd 3.63899 osd.202 up 1.00000 1.00000
>> 203 hdd 3.63899 osd.203 up 1.00000 1.00000
>> 204 hdd 3.63899 osd.204 up 1.00000 1.00000
>> 205 hdd 3.63899 osd.205 up 1.00000 1.00000
>> 206 hdd 3.63899 osd.206 up 1.00000 1.00000
>> 207 hdd 3.63899 osd.207 up 1.00000 1.00000
>> 208 hdd 3.63899 osd.208 up 1.00000 1.00000
>> 209 hdd 3.63899 osd.209 up 1.00000 1.00000
>> 210 hdd 3.63899 osd.210 up 1.00000 1.00000
>> 211 hdd 3.63899 osd.211 up 1.00000 1.00000
>> 212 hdd 3.63899 osd.212 up 1.00000 1.00000
>> 213 hdd 3.63899 osd.213 up 1.00000 1.00000
>> 214 hdd 3.63899 osd.214 up 1.00000 1.00000
>> 215 hdd 3.63899 osd.215 up 1.00000 1.00000
>> 216 hdd 3.63899 osd.216 up 1.00000 1.00000
>> 217 hdd 3.63899 osd.217 up 1.00000 1.00000
>> 218 hdd 3.63899 osd.218 up 1.00000 1.00000
>> 219 hdd 3.63899 osd.219 up 1.00000 1.00000
>> 220 hdd 3.63899 osd.220 up 1.00000 1.00000
>> 221 hdd 3.63899 osd.221 up 1.00000 1.00000
>> 222 hdd 3.63899 osd.222 up 1.00000 1.00000
>> 223 hdd 3.63899 osd.223 up 1.00000 1.00000
>> 224 hdd 3.63899 osd.224 up 1.00000 1.00000
>> 225 hdd 3.63899 osd.225 up 1.00000 1.00000
>> 226 hdd 3.63899 osd.226 up 1.00000 1.00000
>> 227 hdd 3.63899 osd.227 up 1.00000 1.00000
>> 228 hdd 3.63899 osd.228 up 1.00000 1.00000
>> 229 hdd 3.63899 osd.229 up 1.00000 1.00000
>>
>> # ceph osd df
>> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
>> 0 ssd 0.21799 1.00000 223G 4305M 219G 1.88 0.04 64
>> 1 ssd 0.21799 1.00000 223G 3353M 220G 1.46 0.03 64
>> 2 ssd 0.21799 1.00000 223G 1836M 221G 0.80 0.02 56
>> 3 ssd 0.21799 1.00000 223G 2276M 221G 0.99 0.02 72
>> 230 ssd 0.90999 1.00000 931G 2775M 928G 0.29 0.01 60
>> 231 ssd 0.90999 1.00000 931G 3279M 928G 0.34 0.01 68
>> 4 hdd 3.64499 1.00000 3726G 1916G 1809G 51.45 1.02 111
>> 5 hdd 3.64499 1.00000 3726G 2144G 1581G 57.54 1.15 124
>> 6 hdd 3.64499 1.00000 3726G 1887G 1838G 50.66 1.01 109
>> 7 hdd 3.64499 1.00000 3726G 1642G 2083G 44.09 0.88 95
>> 8 hdd 3.64499 1.00000 3726G 1969G 1756G 52.86 1.05 114
>> 9 hdd 3.64499 1.00000 3726G 1577G 2148G 42.34 0.84 91
>> 10 hdd 3.64499 1.00000 3726G 2043G 1682G 54.85 1.09 118
>> 11 hdd 3.64499 1.00000 3726G 1957G 1768G 52.54 1.05 113
>> 12 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114
>> 13 hdd 3.64499 1.00000 3726G 1676G 2049G 45.00 0.90 97
>> 14 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102
>> 15 hdd 3.64499 1.00000 3726G 2161G 1564G 58.02 1.15 125
>> 16 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101
>> 17 hdd 3.64499 1.00000 3726G 2022G 1703G 54.29 1.08 117
>> 18 hdd 3.64499 1.00000 3726G 1784G 1941G 47.89 0.95 103
>> 19 hdd 3.64499 1.00000 3726G 1783G 1942G 47.85 0.95 103
>> 20 hdd 3.64499 1.00000 3726G 1731G 1994G 46.47 0.92 100
>> 21 hdd 3.64499 1.00000 3726G 1939G 1786G 52.06 1.04 112
>> 22 hdd 3.64499 1.00000 3726G 2061G 1664G 55.34 1.10 119
>> 23 hdd 3.64499 1.00000 3726G 1538G 2187G 41.29 0.82 89
>> 24 hdd 3.64499 1.00000 3726G 2168G 1557G 58.19 1.16 125
>> 25 hdd 3.64499 1.00000 3726G 2023G 1702G 54.32 1.08 117
>> 26 hdd 3.64499 1.00000 3726G 1900G 1825G 51.00 1.02 110
>> 27 hdd 3.64499 1.00000 3726G 2126G 1599G 57.08 1.14 123
>> 28 hdd 3.64499 1.00000 3726G 1938G 1787G 52.02 1.04 112
>> 29 hdd 3.64499 1.00000 3726G 1663G 2062G 44.65 0.89 96
>> 30 hdd 3.64499 1.00000 3726G 2163G 1562G 58.07 1.16 125
>> 31 hdd 3.64499 1.00000 3726G 1919G 1806G 51.52 1.03 111
>> 32 hdd 3.64499 1.00000 3726G 1696G 2029G 45.52 0.91 98
>> 33 hdd 3.64499 1.00000 3726G 1438G 2287G 38.61 0.77 83
>> 34 hdd 3.64499 1.00000 3726G 2073G 1652G 55.64 1.11 120
>> 35 hdd 3.64499 1.00000 3726G 2079G 1646G 55.81 1.11 120
>> 36 hdd 3.64499 1.00000 3726G 1955G 1770G 52.50 1.04 113
>> 37 hdd 3.64499 1.00000 3726G 2180G 1545G 58.51 1.16 126
>> 38 hdd 3.64499 1.00000 3726G 1664G 2061G 44.66 0.89 96
>> 39 hdd 3.64499 1.00000 3726G 2003G 1722G 53.77 1.07 116
>> 40 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103
>> 41 hdd 3.64499 1.00000 3726G 1903G 1822G 51.08 1.02 110
>> 42 hdd 3.64499 1.00000 3726G 1747G 1978G 46.89 0.93 101
>> 43 hdd 3.64499 1.00000 3726G 2021G 1704G 54.26 1.08 117
>> 44 hdd 3.64499 1.00000 3726G 1852G 1873G 49.71 0.99 107
>> 45 hdd 3.64499 1.00000 3726G 1522G 2203G 40.87 0.81 88
>> 46 hdd 3.64499 1.00000 3726G 1851G 1874G 49.69 0.99 107
>> 47 hdd 3.64499 1.00000 3726G 1698G 2027G 45.59 0.91 98
>> 48 hdd 3.64499 1.00000 3726G 1940G 1785G 52.08 1.04 112
>> 49 hdd 3.64499 1.00000 3726G 1746G 1979G 46.88 0.93 101
>> 50 hdd 3.64499 1.00000 3726G 1728G 1997G 46.40 0.92 100
>> 51 hdd 3.64499 1.00000 3726G 2093G 1632G 56.18 1.12 121
>> 52 hdd 3.64499 1.00000 3726G 2235G 1490G 60.01 1.19 129
>> 53 hdd 3.64499 1.00000 3726G 1871G 1854G 50.23 1.00 108
>> 54 hdd 3.64499 1.00000 3726G 2094G 1631G 56.22 1.12 121
>> 55 hdd 3.64499 1.00000 3726G 2422G 1303G 65.02 1.29 140
>> 56 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
>> 57 hdd 3.64499 1.00000 3726G 1729G 1996G 46.43 0.92 100
>> 58 hdd 3.64499 1.00000 3726G 1976G 1749G 53.05 1.06 114
>> 59 hdd 3.64499 1.00000 3726G 1954G 1771G 52.46 1.04 113
>> 60 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107
>> 61 hdd 3.64499 1.00000 3726G 2042G 1683G 54.82 1.09 118
>> 62 hdd 3.64499 1.00000 3726G 1921G 1804G 51.58 1.03 111
>> 63 hdd 3.64499 1.00000 3726G 2128G 1597G 57.13 1.14 123
>> 64 hdd 3.64499 1.00000 3726G 2037G 1688G 54.67 1.09 118
>> 65 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114
>> 66 hdd 3.64499 1.00000 3726G 1539G 2186G 41.32 0.82 89
>> 67 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
>> 68 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
>> 69 hdd 3.64499 1.00000 3726G 2092G 1633G 56.16 1.12 121
>> 70 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104
>> 71 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129
>> 72 hdd 3.64499 1.00000 3726G 1557G 2168G 41.79 0.83 90
>> 73 hdd 3.64499 1.00000 3726G 2300G 1425G 61.75 1.23 133
>> 74 hdd 3.64499 1.00000 3726G 2132G 1593G 57.24 1.14 123
>> 75 hdd 3.64499 1.00000 3726G 1818G 1907G 48.80 0.97 105
>> 76 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129
>> 77 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101
>> 78 hdd 3.64499 1.00000 3726G 1766G 1959G 47.41 0.94 102
>> 79 hdd 3.64499 1.00000 3726G 1678G 2047G 45.04 0.90 97
>> 80 hdd 3.63899 1.00000 3726G 1744G 1981G 46.82 0.93 102
>> 81 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
>> 82 hdd 3.64499 1.00000 3726G 1711G 2014G 45.93 0.91 99
>> 83 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100
>> 84 hdd 3.64499 1.00000 3726G 1834G 1891G 49.22 0.98 106
>> 85 hdd 3.64499 1.00000 3726G 1709G 2016G 45.88 0.91 99
>> 86 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112
>> 87 hdd 3.64499 1.00000 3726G 2131G 1594G 57.19 1.14 123
>> 88 hdd 3.64499 1.00000 3726G 1559G 2166G 41.85 0.83 90
>> 89 hdd 3.64499 1.00000 3726G 1645G 2080G 44.16 0.88 95
>> 90 hdd 3.64499 1.00000 3726G 2045G 1680G 54.89 1.09 118
>> 91 hdd 3.64499 1.00000 3726G 2164G 1561G 58.09 1.16 125
>> 92 hdd 3.64499 1.00000 3726G 1922G 1803G 51.59 1.03 111
>> 93 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112
>> 94 hdd 3.64499 1.00000 3726G 1971G 1754G 52.91 1.05 114
>> 95 hdd 3.64499 1.00000 3726G 1917G 1808G 51.47 1.02 111
>> 96 hdd 3.64499 1.00000 3726G 1814G 1911G 48.70 0.97 105
>> 97 hdd 3.64499 1.00000 3726G 1660G 2065G 44.57 0.89 96
>> 98 hdd 3.64499 1.00000 3726G 1904G 1821G 51.10 1.02 110
>> 99 hdd 3.64499 1.00000 3726G 2112G 1613G 56.70 1.13 122
>> 100 hdd 3.64499 1.00000 3726G 1869G 1856G 50.18 1.00 108
>> 101 hdd 3.64499 1.00000 3726G 2005G 1720G 53.84 1.07 116
>> 102 hdd 3.64499 1.00000 3726G 2039G 1686G 54.73 1.09 118
>> 103 hdd 3.64499 1.00000 3726G 2093G 1632G 56.20 1.12 121
>> 104 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
>> 105 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
>> 106 hdd 3.64499 1.00000 3726G 1956G 1769G 52.51 1.05 113
>> 107 hdd 3.64499 1.00000 3726G 2077G 1648G 55.74 1.11 120
>> 108 hdd 3.64499 1.00000 3726G 1990G 1735G 53.43 1.06 115
>> 109 hdd 3.64499 1.00000 3726G 1781G 1944G 47.80 0.95 103
>> 110 hdd 3.64499 1.00000 3726G 1869G 1856G 50.17 1.00 108
>> 111 hdd 3.64499 1.00000 3726G 2023G 1702G 54.30 1.08 117
>> 112 hdd 3.64499 1.00000 3726G 2007G 1718G 53.89 1.07 116
>> 113 hdd 3.64499 1.00000 3726G 1851G 1874G 49.70 0.99 107
>> 114 hdd 3.64499 1.00000 3726G 1832G 1893G 49.18 0.98 106
>> 115 hdd 3.64499 1.00000 3726G 2098G 1627G 56.32 1.12 121
>> 116 hdd 3.64499 1.00000 3726G 1866G 1859G 50.10 1.00 108
>> 117 hdd 3.64499 1.00000 3726G 1764G 1961G 47.36 0.94 102
>> 118 hdd 3.64499 1.00000 3726G 1816G 1909G 48.76 0.97 105
>> 119 hdd 3.64499 1.00000 3726G 1733G 1992G 46.53 0.93 100
>> 120 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
>> 121 hdd 3.64499 1.00000 3726G 1657G 2068G 44.48 0.89 96
>> 122 hdd 3.64499 1.00000 3726G 1715G 2010G 46.04 0.92 99
>> 123 hdd 3.64499 1.00000 3726G 1974G 1751G 53.00 1.05 114
>> 124 hdd 3.64499 1.00000 3726G 1731G 1994G 46.48 0.93 100
>> 125 hdd 3.64499 1.00000 3726G 1897G 1828G 50.94 1.01 110
>> 126 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115
>> 127 hdd 3.64499 1.00000 3726G 1972G 1753G 52.93 1.05 114
>> 128 hdd 3.64499 1.00000 3726G 1888G 1837G 50.68 1.01 109
>> 129 hdd 3.64499 1.00000 3726G 2025G 1700G 54.35 1.08 117
>> 130 hdd 3.64499 1.00000 3726G 1747G 1978G 46.90 0.93 101
>> 131 hdd 3.64499 1.00000 3726G 2061G 1664G 55.32 1.10 119
>> 132 hdd 3.64499 1.00000 3726G 2185G 1540G 58.65 1.17 126
>> 133 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102
>> 134 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103
>> 135 hdd 3.64499 1.00000 3726G 1675G 2050G 44.96 0.89 97
>> 136 hdd 3.64499 1.00000 3726G 1889G 1836G 50.70 1.01 109
>> 137 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113
>> 138 hdd 3.64499 1.00000 3726G 1694G 2031G 45.49 0.91 98
>> 139 hdd 3.64499 1.00000 3726G 2114G 1611G 56.76 1.13 122
>> 140 hdd 3.64499 1.00000 3726G 1726G 1999G 46.35 0.92 100
>> 141 hdd 3.64499 1.00000 3726G 1988G 1737G 53.36 1.06 115
>> 142 hdd 3.64499 1.00000 3726G 2181G 1544G 58.55 1.17 126
>> 143 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107
>> 144 hdd 3.64499 1.00000 3726G 2144G 1581G 57.56 1.15 124
>> 145 hdd 3.64499 1.00000 3726G 1868G 1857G 50.14 1.00 108
>> 146 hdd 3.64499 1.00000 3726G 2078G 1647G 55.77 1.11 120
>> 147 hdd 3.64499 1.00000 3726G 1992G 1733G 53.49 1.06 115
>> 148 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100
>> 149 hdd 3.64499 1.00000 3726G 2162G 1563G 58.03 1.15 125
>> 150 hdd 3.64499 1.00000 3726G 2005G 1720G 53.83 1.07 116
>> 151 hdd 3.64499 1.00000 3726G 1918G 1807G 51.49 1.02 111
>> 152 hdd 3.64499 1.00000 3726G 1921G 1804G 51.57 1.03 111
>> 153 hdd 3.64499 1.00000 3726G 1783G 1942G 47.86 0.95 103
>> 154 hdd 3.64499 1.00000 3726G 2005G 1720G 53.82 1.07 116
>> 155 hdd 3.64499 1.00000 3726G 1853G 1873G 49.73 0.99 107
>> 156 hdd 3.64499 1.00000 3726G 1626G 2099G 43.65 0.87 94
>> 157 hdd 3.64499 1.00000 3726G 1802G 1923G 48.38 0.96 104
>> 158 hdd 3.64499 1.00000 3726G 1765G 1960G 47.39 0.94 102
>> 159 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115
>> 160 hdd 3.64499 1.00000 3726G 1901G 1824G 51.02 1.02 110
>> 161 hdd 3.64499 1.00000 3726G 1906G 1819G 51.16 1.02 110
>> 162 hdd 3.64499 1.00000 3726G 1763G 1962G 47.33 0.94 102
>> 163 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104
>> 164 hdd 3.64499 1.00000 3726G 1922G 1803G 51.61 1.03 111
>> 165 hdd 3.64499 1.00000 3726G 1731G 1994G 46.46 0.92 100
>> 166 hdd 3.64499 1.00000 3726G 2027G 1699G 54.40 1.08 117
>> 167 hdd 3.64499 1.00000 3726G 1956G 1769G 52.50 1.04 113
>> 168 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107
>> 169 hdd 3.64499 1.00000 3726G 1884G 1841G 50.59 1.01 109
>> 170 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
>> 171 hdd 3.64499 1.00000 3726G 1898G 1827G 50.96 1.01 110
>> 172 hdd 3.64499 1.00000 3726G 1781G 1944G 47.82 0.95 103
>> 173 hdd 3.64499 1.00000 3726G 1628G 2097G 43.71 0.87 94
>> 174 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
>> 175 hdd 3.64499 1.00000 3726G 1905G 1820G 51.15 1.02 110
>> 176 hdd 3.64499 1.00000 3726G 1709G 2016G 45.89 0.91 99
>> 177 hdd 3.64499 1.00000 3726G 1991G 1734G 53.44 1.06 115
>> 178 hdd 3.64499 1.00000 3726G 1973G 1752G 52.97 1.05 114
>> 179 hdd 3.64499 1.00000 3726G 1574G 2151G 42.26 0.84 91
>> 180 hdd 3.64499 1.00000 3726G 1917G 1808G 51.46 1.02 111
>> 181 hdd 3.64499 1.00000 3726G 1937G 1788G 52.00 1.03 112
>> 182 hdd 3.64499 1.00000 3726G 1989G 1736G 53.40 1.06 115
>> 183 hdd 3.64499 1.00000 3726G 2039G 1686G 54.74 1.09 118
>> 184 hdd 3.64499 1.00000 3726G 1868G 1857G 50.16 1.00 108
>> 185 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113
>> 186 hdd 3.64499 1.00000 3726G 1850G 1875G 49.66 0.99 107
>> 187 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107
>> 188 hdd 3.64499 1.00000 3726G 1814G 1911G 48.69 0.97 105
>> 189 hdd 3.64499 1.00000 3726G 2286G 1439G 61.36 1.22 132
>> 190 hdd 3.64499 1.00000 3726G 2060G 1665G 55.31 1.10 119
>> 191 hdd 3.64499 1.00000 3726G 1836G 1889G 49.29 0.98 106
>> 192 hdd 3.63869 1.00000 3726G 1721G 2004G 46.21 0.92 101
>> 193 hdd 3.64499 1.00000 3726G 1953G 1772G 52.43 1.04 113
>> 194 hdd 3.64499 1.00000 3726G 1541G 2184G 41.37 0.82 89
>> 195 hdd 3.64499 1.00000 3726G 1837G 1888G 49.32 0.98 106
>> 196 hdd 3.63899 1.00000 3726G 1705G 2020G 45.79 0.91 100
>> 197 hdd 3.63899 1.00000 3726G 1743G 1982G 46.78 0.93 102
>> 198 hdd 3.63899 1.00000 3726G 1982G 1743G 53.21 1.06 116
>> 199 hdd 3.63899 1.00000 3726G 1382G 2343G 37.11 0.74 81
>> 200 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111
>> 201 hdd 3.63899 1.00000 3726G 1845G 1880G 49.54 0.99 108
>> 202 hdd 3.63899 1.00000 3726G 1658G 2067G 44.51 0.89 97
>> 203 hdd 3.63899 1.00000 3726G 2066G 1659G 55.47 1.10 121
>> 204 hdd 3.63899 1.00000 3726G 1672G 2053G 44.90 0.89 98
>> 205 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89
>> 206 hdd 3.63899 1.00000 3726G 1796G 1929G 48.22 0.96 105
>> 207 hdd 3.63899 1.00000 3726G 1555G 2170G 41.75 0.83 91
>> 208 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102
>> 209 hdd 3.63899 1.00000 3726G 2016G 1709G 54.11 1.08 118
>> 210 hdd 3.63899 1.00000 3726G 2136G 1589G 57.34 1.14 125
>> 211 hdd 3.63899 1.00000 3726G 1656G 2069G 44.46 0.88 97
>> 212 hdd 3.63899 1.00000 3726G 1553G 2172G 41.71 0.83 91
>> 213 hdd 3.63899 1.00000 3726G 1758G 1967G 47.19 0.94 103
>> 214 hdd 3.63899 1.00000 3726G 1603G 2122G 43.03 0.86 94
>> 215 hdd 3.63899 1.00000 3726G 1809G 1916G 48.57 0.97 106
>> 216 hdd 3.63899 1.00000 3726G 1760G 1965G 47.26 0.94 103
>> 217 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102
>> 218 hdd 3.63899 1.00000 3726G 1880G 1845G 50.46 1.00 110
>> 219 hdd 3.63899 1.00000 3726G 1693G 2032G 45.45 0.90 99
>> 220 hdd 3.63899 1.00000 3726G 2032G 1693G 54.56 1.09 119
>> 221 hdd 3.63899 1.00000 3726G 1842G 1883G 49.44 0.98 108
>> 222 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89
>> 223 hdd 3.63899 1.00000 3726G 1470G 2255G 39.46 0.79 86
>> 224 hdd 3.63899 1.00000 3726G 1808G 1917G 48.53 0.97 106
>> 225 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111
>> 226 hdd 3.63899 1.00000 3726G 1776G 1949G 47.69 0.95 104
>> 227 hdd 3.63899 1.00000 3726G 1929G 1796G 51.78 1.03 113
>> 228 hdd 3.63899 1.00000 3726G 1657G 2068G 44.48 0.89 97
>> 229 hdd 3.63899 1.00000 3726G 1843G 1882G 49.47 0.98 108
>> TOTAL 825T 414T 410T 50.24
>> MIN/MAX VAR: 0.01/1.29 STDDEV: 9.22
>> --------------------------------------------------------------
>>
>> Am 20.10.18 um 20:35 schrieb Janne Johansson:
>>> Yes, if you have uneven sizes I guess you could end up in a situation
>>> where you have
>>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces
>>> the pool to have one
>>> PG replica on the 1TB OSD, then it would be possible to state "this
>>> pool cant write more than X G"
>>> but when it is full, there would be free space left on some of the
>>> 2TB-OSDs, but which the pool
>>> cant utilize. Probably same for uneven OSD hosts if you have those.
>>>
>>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth
>>> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>>>
>>>> Dear Janne,
>>>>
>>>> yes, of course. But since we only have two pools here, this can not explain the difference.
>>>> The metadata is replicated (3 copies) across ssd drives, and we have < 3 TB of total raw storage for that.
>>>> So looking at the raw space usage, we can ignore that.
>>>>
>>>> All the rest is used for the ceph_data pool. So the ceph_data pool, in terms of raw storage, is about 50 % used.
>>>>
>>>> But in terms of storage shown for that pool, it's almost 63 % %USED.
>>>> So I guess this can purely be from bad balancing, correct?
>>>>
>>>> Cheers,
>>>> Oliver
>>>>
>>>> Am 20.10.18 um 19:49 schrieb Janne Johansson:
>>>>> Do mind that drives may have more than one pool on them, so RAW space
>>>>> is what it says, how much free space there is. Then the avail and
>>>>> %USED on per-pool stats will take replication into account, it can
>>>>> tell how much data you may write into that particular pool, given that
>>>>> pools replication or EC settings.
>>>>>
>>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth
>>>>> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>>>>>
>>>>>> Dear Cephalopodians,
>>>>>>
>>>>>> as many others, I'm also a bit confused by "ceph df" output
>>>>>> in a pretty straightforward configuration.
>>>>>>
>>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile.
>>>>>>
>>>>>> I get:
>>>>>> ----------------------------------------------------------------------------
>>>>>> # ceph df
>>>>>> GLOBAL:
>>>>>> SIZE AVAIL RAW USED %RAW USED
>>>>>> 824T 410T 414T 50.26
>>>>>> POOLS:
>>>>>> NAME ID USED %USED MAX AVAIL OBJECTS
>>>>>> cephfs_metadata 1 452M 0.05 860G 365774
>>>>>> cephfs_data 2 275T 62.68 164T 75056403
>>>>>> ----------------------------------------------------------------------------
>>>>>>
>>>>>> So about 50 % of raw space are used, but already ~63 % of filesystem space are used.
>>>>>> Is this purely from imperfect balancing?
>>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % usage down to 37.12 %.
>>>>>>
>>>>>> We did not yet use the balancer plugin.
>>>>>> We don't have any pre-luminous clients.
>>>>>> In that setup, I take it that "upmap" mode would be recommended - correct?
>>>>>> Any "gotchas" using that on luminous?
>>>>>>
>>>>>> Cheers,
>>>>>> Oliver
>>>>>>
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list
>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
