All OSDs are of the very same size. One OSD host has slightly more disks (33 instead of 31), though.
So also that that can't explain the hefty difference.
I attach the output of "ceph osd tree" and "ceph osd df".
The crush rule for the ceph_data pool is:
rule cephfs_data {
id 2
type erasure
min_size 3
max_size 6
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step chooseleaf indep 0 type host
step emit
}
So that only considers the hdd device class. EC is done with k=4 m=2.
So I don't see any imbalance on the hardware level, but only a somewhat uneven distribution of PGs.
Am I missing something, or is this really just a case for the ceph balancer plugin?
I'm just a bit astonished this effect is so huge.
Maybe our 4096 PGs for the ceph_data pool are not enough to get an even distribution without balancing?
But it yields about 100 PGs per OSD, as you can see...
--------------------------------------------------------------
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 826.26428 root default
-3 0.43700 host mon001
0 ssd 0.21799 osd.0 up 1.00000 1.00000
1 ssd 0.21799 osd.1 up 1.00000 1.00000
-5 0.43700 host mon002
2 ssd 0.21799 osd.2 up 1.00000 1.00000
3 ssd 0.21799 osd.3 up 1.00000 1.00000
-31 1.81898 host mon003
230 ssd 0.90999 osd.230 up 1.00000 1.00000
231 ssd 0.90999 osd.231 up 1.00000 1.00000
-10 116.64600 host osd001
4 hdd 3.64499 osd.4 up 1.00000 1.00000
5 hdd 3.64499 osd.5 up 1.00000 1.00000
6 hdd 3.64499 osd.6 up 1.00000 1.00000
7 hdd 3.64499 osd.7 up 1.00000 1.00000
8 hdd 3.64499 osd.8 up 1.00000 1.00000
9 hdd 3.64499 osd.9 up 1.00000 1.00000
10 hdd 3.64499 osd.10 up 1.00000 1.00000
11 hdd 3.64499 osd.11 up 1.00000 1.00000
12 hdd 3.64499 osd.12 up 1.00000 1.00000
13 hdd 3.64499 osd.13 up 1.00000 1.00000
14 hdd 3.64499 osd.14 up 1.00000 1.00000
15 hdd 3.64499 osd.15 up 1.00000 1.00000
16 hdd 3.64499 osd.16 up 1.00000 1.00000
17 hdd 3.64499 osd.17 up 1.00000 1.00000
18 hdd 3.64499 osd.18 up 1.00000 1.00000
19 hdd 3.64499 osd.19 up 1.00000 1.00000
20 hdd 3.64499 osd.20 up 1.00000 1.00000
21 hdd 3.64499 osd.21 up 1.00000 1.00000
22 hdd 3.64499 osd.22 up 1.00000 1.00000
23 hdd 3.64499 osd.23 up 1.00000 1.00000
24 hdd 3.64499 osd.24 up 1.00000 1.00000
25 hdd 3.64499 osd.25 up 1.00000 1.00000
26 hdd 3.64499 osd.26 up 1.00000 1.00000
27 hdd 3.64499 osd.27 up 1.00000 1.00000
28 hdd 3.64499 osd.28 up 1.00000 1.00000
29 hdd 3.64499 osd.29 up 1.00000 1.00000
30 hdd 3.64499 osd.30 up 1.00000 1.00000
31 hdd 3.64499 osd.31 up 1.00000 1.00000
32 hdd 3.64499 osd.32 up 1.00000 1.00000
33 hdd 3.64499 osd.33 up 1.00000 1.00000
34 hdd 3.64499 osd.34 up 1.00000 1.00000
35 hdd 3.64499 osd.35 up 1.00000 1.00000
-13 116.64600 host osd002
36 hdd 3.64499 osd.36 up 1.00000 1.00000
37 hdd 3.64499 osd.37 up 1.00000 1.00000
38 hdd 3.64499 osd.38 up 1.00000 1.00000
39 hdd 3.64499 osd.39 up 1.00000 1.00000
40 hdd 3.64499 osd.40 up 1.00000 1.00000
41 hdd 3.64499 osd.41 up 1.00000 1.00000
42 hdd 3.64499 osd.42 up 1.00000 1.00000
43 hdd 3.64499 osd.43 up 1.00000 1.00000
44 hdd 3.64499 osd.44 up 1.00000 1.00000
45 hdd 3.64499 osd.45 up 1.00000 1.00000
46 hdd 3.64499 osd.46 up 1.00000 1.00000
47 hdd 3.64499 osd.47 up 1.00000 1.00000
48 hdd 3.64499 osd.48 up 1.00000 1.00000
49 hdd 3.64499 osd.49 up 1.00000 1.00000
50 hdd 3.64499 osd.50 up 1.00000 1.00000
51 hdd 3.64499 osd.51 up 1.00000 1.00000
52 hdd 3.64499 osd.52 up 1.00000 1.00000
53 hdd 3.64499 osd.53 up 1.00000 1.00000
54 hdd 3.64499 osd.54 up 1.00000 1.00000
55 hdd 3.64499 osd.55 up 1.00000 1.00000
56 hdd 3.64499 osd.56 up 1.00000 1.00000
57 hdd 3.64499 osd.57 up 1.00000 1.00000
58 hdd 3.64499 osd.58 up 1.00000 1.00000
59 hdd 3.64499 osd.59 up 1.00000 1.00000
60 hdd 3.64499 osd.60 up 1.00000 1.00000
61 hdd 3.64499 osd.61 up 1.00000 1.00000
62 hdd 3.64499 osd.62 up 1.00000 1.00000
63 hdd 3.64499 osd.63 up 1.00000 1.00000
64 hdd 3.64499 osd.64 up 1.00000 1.00000
65 hdd 3.64499 osd.65 up 1.00000 1.00000
66 hdd 3.64499 osd.66 up 1.00000 1.00000
67 hdd 3.64499 osd.67 up 1.00000 1.00000
-16 116.63899 host osd003
68 hdd 3.64499 osd.68 up 1.00000 1.00000
69 hdd 3.64499 osd.69 up 1.00000 1.00000
70 hdd 3.64499 osd.70 up 1.00000 1.00000
71 hdd 3.64499 osd.71 up 1.00000 1.00000
72 hdd 3.64499 osd.72 up 1.00000 1.00000
73 hdd 3.64499 osd.73 up 1.00000 1.00000
74 hdd 3.64499 osd.74 up 1.00000 1.00000
75 hdd 3.64499 osd.75 up 1.00000 1.00000
76 hdd 3.64499 osd.76 up 1.00000 1.00000
77 hdd 3.64499 osd.77 up 1.00000 1.00000
78 hdd 3.64499 osd.78 up 1.00000 1.00000
79 hdd 3.64499 osd.79 up 1.00000 1.00000
80 hdd 3.63899 osd.80 up 1.00000 1.00000
81 hdd 3.64499 osd.81 up 1.00000 1.00000
82 hdd 3.64499 osd.82 up 1.00000 1.00000
83 hdd 3.64499 osd.83 up 1.00000 1.00000
84 hdd 3.64499 osd.84 up 1.00000 1.00000
85 hdd 3.64499 osd.85 up 1.00000 1.00000
86 hdd 3.64499 osd.86 up 1.00000 1.00000
87 hdd 3.64499 osd.87 up 1.00000 1.00000
88 hdd 3.64499 osd.88 up 1.00000 1.00000
89 hdd 3.64499 osd.89 up 1.00000 1.00000
90 hdd 3.64499 osd.90 up 1.00000 1.00000
91 hdd 3.64499 osd.91 up 1.00000 1.00000
92 hdd 3.64499 osd.92 up 1.00000 1.00000
93 hdd 3.64499 osd.93 up 1.00000 1.00000
94 hdd 3.64499 osd.94 up 1.00000 1.00000
95 hdd 3.64499 osd.95 up 1.00000 1.00000
96 hdd 3.64499 osd.96 up 1.00000 1.00000
97 hdd 3.64499 osd.97 up 1.00000 1.00000
98 hdd 3.64499 osd.98 up 1.00000 1.00000
99 hdd 3.64499 osd.99 up 1.00000 1.00000
-19 116.64600 host osd004
100 hdd 3.64499 osd.100 up 1.00000 1.00000
101 hdd 3.64499 osd.101 up 1.00000 1.00000
102 hdd 3.64499 osd.102 up 1.00000 1.00000
103 hdd 3.64499 osd.103 up 1.00000 1.00000
104 hdd 3.64499 osd.104 up 1.00000 1.00000
105 hdd 3.64499 osd.105 up 1.00000 1.00000
106 hdd 3.64499 osd.106 up 1.00000 1.00000
107 hdd 3.64499 osd.107 up 1.00000 1.00000
108 hdd 3.64499 osd.108 up 1.00000 1.00000
109 hdd 3.64499 osd.109 up 1.00000 1.00000
110 hdd 3.64499 osd.110 up 1.00000 1.00000
111 hdd 3.64499 osd.111 up 1.00000 1.00000
112 hdd 3.64499 osd.112 up 1.00000 1.00000
113 hdd 3.64499 osd.113 up 1.00000 1.00000
114 hdd 3.64499 osd.114 up 1.00000 1.00000
115 hdd 3.64499 osd.115 up 1.00000 1.00000
116 hdd 3.64499 osd.116 up 1.00000 1.00000
117 hdd 3.64499 osd.117 up 1.00000 1.00000
118 hdd 3.64499 osd.118 up 1.00000 1.00000
119 hdd 3.64499 osd.119 up 1.00000 1.00000
120 hdd 3.64499 osd.120 up 1.00000 1.00000
121 hdd 3.64499 osd.121 up 1.00000 1.00000
122 hdd 3.64499 osd.122 up 1.00000 1.00000
123 hdd 3.64499 osd.123 up 1.00000 1.00000
124 hdd 3.64499 osd.124 up 1.00000 1.00000
125 hdd 3.64499 osd.125 up 1.00000 1.00000
126 hdd 3.64499 osd.126 up 1.00000 1.00000
127 hdd 3.64499 osd.127 up 1.00000 1.00000
128 hdd 3.64499 osd.128 up 1.00000 1.00000
129 hdd 3.64499 osd.129 up 1.00000 1.00000
130 hdd 3.64499 osd.130 up 1.00000 1.00000
131 hdd 3.64499 osd.131 up 1.00000 1.00000
-22 116.64600 host osd005
132 hdd 3.64499 osd.132 up 1.00000 1.00000
133 hdd 3.64499 osd.133 up 1.00000 1.00000
134 hdd 3.64499 osd.134 up 1.00000 1.00000
135 hdd 3.64499 osd.135 up 1.00000 1.00000
136 hdd 3.64499 osd.136 up 1.00000 1.00000
137 hdd 3.64499 osd.137 up 1.00000 1.00000
138 hdd 3.64499 osd.138 up 1.00000 1.00000
139 hdd 3.64499 osd.139 up 1.00000 1.00000
140 hdd 3.64499 osd.140 up 1.00000 1.00000
141 hdd 3.64499 osd.141 up 1.00000 1.00000
142 hdd 3.64499 osd.142 up 1.00000 1.00000
143 hdd 3.64499 osd.143 up 1.00000 1.00000
144 hdd 3.64499 osd.144 up 1.00000 1.00000
145 hdd 3.64499 osd.145 up 1.00000 1.00000
146 hdd 3.64499 osd.146 up 1.00000 1.00000
147 hdd 3.64499 osd.147 up 1.00000 1.00000
148 hdd 3.64499 osd.148 up 1.00000 1.00000
149 hdd 3.64499 osd.149 up 1.00000 1.00000
150 hdd 3.64499 osd.150 up 1.00000 1.00000
151 hdd 3.64499 osd.151 up 1.00000 1.00000
152 hdd 3.64499 osd.152 up 1.00000 1.00000
153 hdd 3.64499 osd.153 up 1.00000 1.00000
154 hdd 3.64499 osd.154 up 1.00000 1.00000
155 hdd 3.64499 osd.155 up 1.00000 1.00000
156 hdd 3.64499 osd.156 up 1.00000 1.00000
157 hdd 3.64499 osd.157 up 1.00000 1.00000
158 hdd 3.64499 osd.158 up 1.00000 1.00000
159 hdd 3.64499 osd.159 up 1.00000 1.00000
160 hdd 3.64499 osd.160 up 1.00000 1.00000
161 hdd 3.64499 osd.161 up 1.00000 1.00000
162 hdd 3.64499 osd.162 up 1.00000 1.00000
163 hdd 3.64499 osd.163 up 1.00000 1.00000
-25 116.63335 host osd006
164 hdd 3.64499 osd.164 up 1.00000 1.00000
165 hdd 3.64499 osd.165 up 1.00000 1.00000
166 hdd 3.64499 osd.166 up 1.00000 1.00000
167 hdd 3.64499 osd.167 up 1.00000 1.00000
168 hdd 3.64499 osd.168 up 1.00000 1.00000
169 hdd 3.64499 osd.169 up 1.00000 1.00000
170 hdd 3.64499 osd.170 up 1.00000 1.00000
171 hdd 3.64499 osd.171 up 1.00000 1.00000
172 hdd 3.64499 osd.172 up 1.00000 1.00000
173 hdd 3.64499 osd.173 up 1.00000 1.00000
174 hdd 3.64499 osd.174 up 1.00000 1.00000
175 hdd 3.64499 osd.175 up 1.00000 1.00000
176 hdd 3.64499 osd.176 up 1.00000 1.00000
177 hdd 3.64499 osd.177 up 1.00000 1.00000
178 hdd 3.64499 osd.178 up 1.00000 1.00000
179 hdd 3.64499 osd.179 up 1.00000 1.00000
180 hdd 3.64499 osd.180 up 1.00000 1.00000
181 hdd 3.64499 osd.181 up 1.00000 1.00000
182 hdd 3.64499 osd.182 up 1.00000 1.00000
183 hdd 3.64499 osd.183 up 1.00000 1.00000
184 hdd 3.64499 osd.184 up 1.00000 1.00000
185 hdd 3.64499 osd.185 up 1.00000 1.00000
186 hdd 3.64499 osd.186 up 1.00000 1.00000
187 hdd 3.64499 osd.187 up 1.00000 1.00000
188 hdd 3.64499 osd.188 up 1.00000 1.00000
189 hdd 3.64499 osd.189 up 1.00000 1.00000
190 hdd 3.64499 osd.190 up 1.00000 1.00000
191 hdd 3.64499 osd.191 up 1.00000 1.00000
192 hdd 3.63869 osd.192 up 1.00000 1.00000
193 hdd 3.64499 osd.193 up 1.00000 1.00000
194 hdd 3.64499 osd.194 up 1.00000 1.00000
195 hdd 3.64499 osd.195 up 1.00000 1.00000
-28 123.71500 host osd007
196 hdd 3.63899 osd.196 up 1.00000 1.00000
197 hdd 3.63899 osd.197 up 1.00000 1.00000
198 hdd 3.63899 osd.198 up 1.00000 1.00000
199 hdd 3.63899 osd.199 up 1.00000 1.00000
200 hdd 3.63899 osd.200 up 1.00000 1.00000
201 hdd 3.63899 osd.201 up 1.00000 1.00000
202 hdd 3.63899 osd.202 up 1.00000 1.00000
203 hdd 3.63899 osd.203 up 1.00000 1.00000
204 hdd 3.63899 osd.204 up 1.00000 1.00000
205 hdd 3.63899 osd.205 up 1.00000 1.00000
206 hdd 3.63899 osd.206 up 1.00000 1.00000
207 hdd 3.63899 osd.207 up 1.00000 1.00000
208 hdd 3.63899 osd.208 up 1.00000 1.00000
209 hdd 3.63899 osd.209 up 1.00000 1.00000
210 hdd 3.63899 osd.210 up 1.00000 1.00000
211 hdd 3.63899 osd.211 up 1.00000 1.00000
212 hdd 3.63899 osd.212 up 1.00000 1.00000
213 hdd 3.63899 osd.213 up 1.00000 1.00000
214 hdd 3.63899 osd.214 up 1.00000 1.00000
215 hdd 3.63899 osd.215 up 1.00000 1.00000
216 hdd 3.63899 osd.216 up 1.00000 1.00000
217 hdd 3.63899 osd.217 up 1.00000 1.00000
218 hdd 3.63899 osd.218 up 1.00000 1.00000
219 hdd 3.63899 osd.219 up 1.00000 1.00000
220 hdd 3.63899 osd.220 up 1.00000 1.00000
221 hdd 3.63899 osd.221 up 1.00000 1.00000
222 hdd 3.63899 osd.222 up 1.00000 1.00000
223 hdd 3.63899 osd.223 up 1.00000 1.00000
224 hdd 3.63899 osd.224 up 1.00000 1.00000
225 hdd 3.63899 osd.225 up 1.00000 1.00000
226 hdd 3.63899 osd.226 up 1.00000 1.00000
227 hdd 3.63899 osd.227 up 1.00000 1.00000
228 hdd 3.63899 osd.228 up 1.00000 1.00000
229 hdd 3.63899 osd.229 up 1.00000 1.00000
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 ssd 0.21799 1.00000 223G 4305M 219G 1.88 0.04 64
1 ssd 0.21799 1.00000 223G 3353M 220G 1.46 0.03 64
2 ssd 0.21799 1.00000 223G 1836M 221G 0.80 0.02 56
3 ssd 0.21799 1.00000 223G 2276M 221G 0.99 0.02 72
230 ssd 0.90999 1.00000 931G 2775M 928G 0.29 0.01 60
231 ssd 0.90999 1.00000 931G 3279M 928G 0.34 0.01 68
4 hdd 3.64499 1.00000 3726G 1916G 1809G 51.45 1.02 111
5 hdd 3.64499 1.00000 3726G 2144G 1581G 57.54 1.15 124
6 hdd 3.64499 1.00000 3726G 1887G 1838G 50.66 1.01 109
7 hdd 3.64499 1.00000 3726G 1642G 2083G 44.09 0.88 95
8 hdd 3.64499 1.00000 3726G 1969G 1756G 52.86 1.05 114
9 hdd 3.64499 1.00000 3726G 1577G 2148G 42.34 0.84 91
10 hdd 3.64499 1.00000 3726G 2043G 1682G 54.85 1.09 118
11 hdd 3.64499 1.00000 3726G 1957G 1768G 52.54 1.05 113
12 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114
13 hdd 3.64499 1.00000 3726G 1676G 2049G 45.00 0.90 97
14 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102
15 hdd 3.64499 1.00000 3726G 2161G 1564G 58.02 1.15 125
16 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101
17 hdd 3.64499 1.00000 3726G 2022G 1703G 54.29 1.08 117
18 hdd 3.64499 1.00000 3726G 1784G 1941G 47.89 0.95 103
19 hdd 3.64499 1.00000 3726G 1783G 1942G 47.85 0.95 103
20 hdd 3.64499 1.00000 3726G 1731G 1994G 46.47 0.92 100
21 hdd 3.64499 1.00000 3726G 1939G 1786G 52.06 1.04 112
22 hdd 3.64499 1.00000 3726G 2061G 1664G 55.34 1.10 119
23 hdd 3.64499 1.00000 3726G 1538G 2187G 41.29 0.82 89
24 hdd 3.64499 1.00000 3726G 2168G 1557G 58.19 1.16 125
25 hdd 3.64499 1.00000 3726G 2023G 1702G 54.32 1.08 117
26 hdd 3.64499 1.00000 3726G 1900G 1825G 51.00 1.02 110
27 hdd 3.64499 1.00000 3726G 2126G 1599G 57.08 1.14 123
28 hdd 3.64499 1.00000 3726G 1938G 1787G 52.02 1.04 112
29 hdd 3.64499 1.00000 3726G 1663G 2062G 44.65 0.89 96
30 hdd 3.64499 1.00000 3726G 2163G 1562G 58.07 1.16 125
31 hdd 3.64499 1.00000 3726G 1919G 1806G 51.52 1.03 111
32 hdd 3.64499 1.00000 3726G 1696G 2029G 45.52 0.91 98
33 hdd 3.64499 1.00000 3726G 1438G 2287G 38.61 0.77 83
34 hdd 3.64499 1.00000 3726G 2073G 1652G 55.64 1.11 120
35 hdd 3.64499 1.00000 3726G 2079G 1646G 55.81 1.11 120
36 hdd 3.64499 1.00000 3726G 1955G 1770G 52.50 1.04 113
37 hdd 3.64499 1.00000 3726G 2180G 1545G 58.51 1.16 126
38 hdd 3.64499 1.00000 3726G 1664G 2061G 44.66 0.89 96
39 hdd 3.64499 1.00000 3726G 2003G 1722G 53.77 1.07 116
40 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103
41 hdd 3.64499 1.00000 3726G 1903G 1822G 51.08 1.02 110
42 hdd 3.64499 1.00000 3726G 1747G 1978G 46.89 0.93 101
43 hdd 3.64499 1.00000 3726G 2021G 1704G 54.26 1.08 117
44 hdd 3.64499 1.00000 3726G 1852G 1873G 49.71 0.99 107
45 hdd 3.64499 1.00000 3726G 1522G 2203G 40.87 0.81 88
46 hdd 3.64499 1.00000 3726G 1851G 1874G 49.69 0.99 107
47 hdd 3.64499 1.00000 3726G 1698G 2027G 45.59 0.91 98
48 hdd 3.64499 1.00000 3726G 1940G 1785G 52.08 1.04 112
49 hdd 3.64499 1.00000 3726G 1746G 1979G 46.88 0.93 101
50 hdd 3.64499 1.00000 3726G 1728G 1997G 46.40 0.92 100
51 hdd 3.64499 1.00000 3726G 2093G 1632G 56.18 1.12 121
52 hdd 3.64499 1.00000 3726G 2235G 1490G 60.01 1.19 129
53 hdd 3.64499 1.00000 3726G 1871G 1854G 50.23 1.00 108
54 hdd 3.64499 1.00000 3726G 2094G 1631G 56.22 1.12 121
55 hdd 3.64499 1.00000 3726G 2422G 1303G 65.02 1.29 140
56 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
57 hdd 3.64499 1.00000 3726G 1729G 1996G 46.43 0.92 100
58 hdd 3.64499 1.00000 3726G 1976G 1749G 53.05 1.06 114
59 hdd 3.64499 1.00000 3726G 1954G 1771G 52.46 1.04 113
60 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107
61 hdd 3.64499 1.00000 3726G 2042G 1683G 54.82 1.09 118
62 hdd 3.64499 1.00000 3726G 1921G 1804G 51.58 1.03 111
63 hdd 3.64499 1.00000 3726G 2128G 1597G 57.13 1.14 123
64 hdd 3.64499 1.00000 3726G 2037G 1688G 54.67 1.09 118
65 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114
66 hdd 3.64499 1.00000 3726G 1539G 2186G 41.32 0.82 89
67 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
68 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
69 hdd 3.64499 1.00000 3726G 2092G 1633G 56.16 1.12 121
70 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104
71 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129
72 hdd 3.64499 1.00000 3726G 1557G 2168G 41.79 0.83 90
73 hdd 3.64499 1.00000 3726G 2300G 1425G 61.75 1.23 133
74 hdd 3.64499 1.00000 3726G 2132G 1593G 57.24 1.14 123
75 hdd 3.64499 1.00000 3726G 1818G 1907G 48.80 0.97 105
76 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129
77 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101
78 hdd 3.64499 1.00000 3726G 1766G 1959G 47.41 0.94 102
79 hdd 3.64499 1.00000 3726G 1678G 2047G 45.04 0.90 97
80 hdd 3.63899 1.00000 3726G 1744G 1981G 46.82 0.93 102
81 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
82 hdd 3.64499 1.00000 3726G 1711G 2014G 45.93 0.91 99
83 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100
84 hdd 3.64499 1.00000 3726G 1834G 1891G 49.22 0.98 106
85 hdd 3.64499 1.00000 3726G 1709G 2016G 45.88 0.91 99
86 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112
87 hdd 3.64499 1.00000 3726G 2131G 1594G 57.19 1.14 123
88 hdd 3.64499 1.00000 3726G 1559G 2166G 41.85 0.83 90
89 hdd 3.64499 1.00000 3726G 1645G 2080G 44.16 0.88 95
90 hdd 3.64499 1.00000 3726G 2045G 1680G 54.89 1.09 118
91 hdd 3.64499 1.00000 3726G 2164G 1561G 58.09 1.16 125
92 hdd 3.64499 1.00000 3726G 1922G 1803G 51.59 1.03 111
93 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112
94 hdd 3.64499 1.00000 3726G 1971G 1754G 52.91 1.05 114
95 hdd 3.64499 1.00000 3726G 1917G 1808G 51.47 1.02 111
96 hdd 3.64499 1.00000 3726G 1814G 1911G 48.70 0.97 105
97 hdd 3.64499 1.00000 3726G 1660G 2065G 44.57 0.89 96
98 hdd 3.64499 1.00000 3726G 1904G 1821G 51.10 1.02 110
99 hdd 3.64499 1.00000 3726G 2112G 1613G 56.70 1.13 122
100 hdd 3.64499 1.00000 3726G 1869G 1856G 50.18 1.00 108
101 hdd 3.64499 1.00000 3726G 2005G 1720G 53.84 1.07 116
102 hdd 3.64499 1.00000 3726G 2039G 1686G 54.73 1.09 118
103 hdd 3.64499 1.00000 3726G 2093G 1632G 56.20 1.12 121
104 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109
105 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108
106 hdd 3.64499 1.00000 3726G 1956G 1769G 52.51 1.05 113
107 hdd 3.64499 1.00000 3726G 2077G 1648G 55.74 1.11 120
108 hdd 3.64499 1.00000 3726G 1990G 1735G 53.43 1.06 115
109 hdd 3.64499 1.00000 3726G 1781G 1944G 47.80 0.95 103
110 hdd 3.64499 1.00000 3726G 1869G 1856G 50.17 1.00 108
111 hdd 3.64499 1.00000 3726G 2023G 1702G 54.30 1.08 117
112 hdd 3.64499 1.00000 3726G 2007G 1718G 53.89 1.07 116
113 hdd 3.64499 1.00000 3726G 1851G 1874G 49.70 0.99 107
114 hdd 3.64499 1.00000 3726G 1832G 1893G 49.18 0.98 106
115 hdd 3.64499 1.00000 3726G 2098G 1627G 56.32 1.12 121
116 hdd 3.64499 1.00000 3726G 1866G 1859G 50.10 1.00 108
117 hdd 3.64499 1.00000 3726G 1764G 1961G 47.36 0.94 102
118 hdd 3.64499 1.00000 3726G 1816G 1909G 48.76 0.97 105
119 hdd 3.64499 1.00000 3726G 1733G 1992G 46.53 0.93 100
120 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
121 hdd 3.64499 1.00000 3726G 1657G 2068G 44.48 0.89 96
122 hdd 3.64499 1.00000 3726G 1715G 2010G 46.04 0.92 99
123 hdd 3.64499 1.00000 3726G 1974G 1751G 53.00 1.05 114
124 hdd 3.64499 1.00000 3726G 1731G 1994G 46.48 0.93 100
125 hdd 3.64499 1.00000 3726G 1897G 1828G 50.94 1.01 110
126 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115
127 hdd 3.64499 1.00000 3726G 1972G 1753G 52.93 1.05 114
128 hdd 3.64499 1.00000 3726G 1888G 1837G 50.68 1.01 109
129 hdd 3.64499 1.00000 3726G 2025G 1700G 54.35 1.08 117
130 hdd 3.64499 1.00000 3726G 1747G 1978G 46.90 0.93 101
131 hdd 3.64499 1.00000 3726G 2061G 1664G 55.32 1.10 119
132 hdd 3.64499 1.00000 3726G 2185G 1540G 58.65 1.17 126
133 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102
134 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103
135 hdd 3.64499 1.00000 3726G 1675G 2050G 44.96 0.89 97
136 hdd 3.64499 1.00000 3726G 1889G 1836G 50.70 1.01 109
137 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113
138 hdd 3.64499 1.00000 3726G 1694G 2031G 45.49 0.91 98
139 hdd 3.64499 1.00000 3726G 2114G 1611G 56.76 1.13 122
140 hdd 3.64499 1.00000 3726G 1726G 1999G 46.35 0.92 100
141 hdd 3.64499 1.00000 3726G 1988G 1737G 53.36 1.06 115
142 hdd 3.64499 1.00000 3726G 2181G 1544G 58.55 1.17 126
143 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107
144 hdd 3.64499 1.00000 3726G 2144G 1581G 57.56 1.15 124
145 hdd 3.64499 1.00000 3726G 1868G 1857G 50.14 1.00 108
146 hdd 3.64499 1.00000 3726G 2078G 1647G 55.77 1.11 120
147 hdd 3.64499 1.00000 3726G 1992G 1733G 53.49 1.06 115
148 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100
149 hdd 3.64499 1.00000 3726G 2162G 1563G 58.03 1.15 125
150 hdd 3.64499 1.00000 3726G 2005G 1720G 53.83 1.07 116
151 hdd 3.64499 1.00000 3726G 1918G 1807G 51.49 1.02 111
152 hdd 3.64499 1.00000 3726G 1921G 1804G 51.57 1.03 111
153 hdd 3.64499 1.00000 3726G 1783G 1942G 47.86 0.95 103
154 hdd 3.64499 1.00000 3726G 2005G 1720G 53.82 1.07 116
155 hdd 3.64499 1.00000 3726G 1853G 1873G 49.73 0.99 107
156 hdd 3.64499 1.00000 3726G 1626G 2099G 43.65 0.87 94
157 hdd 3.64499 1.00000 3726G 1802G 1923G 48.38 0.96 104
158 hdd 3.64499 1.00000 3726G 1765G 1960G 47.39 0.94 102
159 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115
160 hdd 3.64499 1.00000 3726G 1901G 1824G 51.02 1.02 110
161 hdd 3.64499 1.00000 3726G 1906G 1819G 51.16 1.02 110
162 hdd 3.64499 1.00000 3726G 1763G 1962G 47.33 0.94 102
163 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104
164 hdd 3.64499 1.00000 3726G 1922G 1803G 51.61 1.03 111
165 hdd 3.64499 1.00000 3726G 1731G 1994G 46.46 0.92 100
166 hdd 3.64499 1.00000 3726G 2027G 1699G 54.40 1.08 117
167 hdd 3.64499 1.00000 3726G 1956G 1769G 52.50 1.04 113
168 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107
169 hdd 3.64499 1.00000 3726G 1884G 1841G 50.59 1.01 109
170 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
171 hdd 3.64499 1.00000 3726G 1898G 1827G 50.96 1.01 110
172 hdd 3.64499 1.00000 3726G 1781G 1944G 47.82 0.95 103
173 hdd 3.64499 1.00000 3726G 1628G 2097G 43.71 0.87 94
174 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115
175 hdd 3.64499 1.00000 3726G 1905G 1820G 51.15 1.02 110
176 hdd 3.64499 1.00000 3726G 1709G 2016G 45.89 0.91 99
177 hdd 3.64499 1.00000 3726G 1991G 1734G 53.44 1.06 115
178 hdd 3.64499 1.00000 3726G 1973G 1752G 52.97 1.05 114
179 hdd 3.64499 1.00000 3726G 1574G 2151G 42.26 0.84 91
180 hdd 3.64499 1.00000 3726G 1917G 1808G 51.46 1.02 111
181 hdd 3.64499 1.00000 3726G 1937G 1788G 52.00 1.03 112
182 hdd 3.64499 1.00000 3726G 1989G 1736G 53.40 1.06 115
183 hdd 3.64499 1.00000 3726G 2039G 1686G 54.74 1.09 118
184 hdd 3.64499 1.00000 3726G 1868G 1857G 50.16 1.00 108
185 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113
186 hdd 3.64499 1.00000 3726G 1850G 1875G 49.66 0.99 107
187 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107
188 hdd 3.64499 1.00000 3726G 1814G 1911G 48.69 0.97 105
189 hdd 3.64499 1.00000 3726G 2286G 1439G 61.36 1.22 132
190 hdd 3.64499 1.00000 3726G 2060G 1665G 55.31 1.10 119
191 hdd 3.64499 1.00000 3726G 1836G 1889G 49.29 0.98 106
192 hdd 3.63869 1.00000 3726G 1721G 2004G 46.21 0.92 101
193 hdd 3.64499 1.00000 3726G 1953G 1772G 52.43 1.04 113
194 hdd 3.64499 1.00000 3726G 1541G 2184G 41.37 0.82 89
195 hdd 3.64499 1.00000 3726G 1837G 1888G 49.32 0.98 106
196 hdd 3.63899 1.00000 3726G 1705G 2020G 45.79 0.91 100
197 hdd 3.63899 1.00000 3726G 1743G 1982G 46.78 0.93 102
198 hdd 3.63899 1.00000 3726G 1982G 1743G 53.21 1.06 116
199 hdd 3.63899 1.00000 3726G 1382G 2343G 37.11 0.74 81
200 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111
201 hdd 3.63899 1.00000 3726G 1845G 1880G 49.54 0.99 108
202 hdd 3.63899 1.00000 3726G 1658G 2067G 44.51 0.89 97
203 hdd 3.63899 1.00000 3726G 2066G 1659G 55.47 1.10 121
204 hdd 3.63899 1.00000 3726G 1672G 2053G 44.90 0.89 98
205 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89
206 hdd 3.63899 1.00000 3726G 1796G 1929G 48.22 0.96 105
207 hdd 3.63899 1.00000 3726G 1555G 2170G 41.75 0.83 91
208 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102
209 hdd 3.63899 1.00000 3726G 2016G 1709G 54.11 1.08 118
210 hdd 3.63899 1.00000 3726G 2136G 1589G 57.34 1.14 125
211 hdd 3.63899 1.00000 3726G 1656G 2069G 44.46 0.88 97
212 hdd 3.63899 1.00000 3726G 1553G 2172G 41.71 0.83 91
213 hdd 3.63899 1.00000 3726G 1758G 1967G 47.19 0.94 103
214 hdd 3.63899 1.00000 3726G 1603G 2122G 43.03 0.86 94
215 hdd 3.63899 1.00000 3726G 1809G 1916G 48.57 0.97 106
216 hdd 3.63899 1.00000 3726G 1760G 1965G 47.26 0.94 103
217 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102
218 hdd 3.63899 1.00000 3726G 1880G 1845G 50.46 1.00 110
219 hdd 3.63899 1.00000 3726G 1693G 2032G 45.45 0.90 99
220 hdd 3.63899 1.00000 3726G 2032G 1693G 54.56 1.09 119
221 hdd 3.63899 1.00000 3726G 1842G 1883G 49.44 0.98 108
222 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89
223 hdd 3.63899 1.00000 3726G 1470G 2255G 39.46 0.79 86
224 hdd 3.63899 1.00000 3726G 1808G 1917G 48.53 0.97 106
225 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111
226 hdd 3.63899 1.00000 3726G 1776G 1949G 47.69 0.95 104
227 hdd 3.63899 1.00000 3726G 1929G 1796G 51.78 1.03 113
228 hdd 3.63899 1.00000 3726G 1657G 2068G 44.48 0.89 97
229 hdd 3.63899 1.00000 3726G 1843G 1882G 49.47 0.98 108
TOTAL 825T 414T 410T 50.24
MIN/MAX VAR: 0.01/1.29 STDDEV: 9.22
--------------------------------------------------------------
Am 20.10.18 um 20:35 schrieb Janne Johansson:
> Yes, if you have uneven sizes I guess you could end up in a situation
> where you have
> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces
> the pool to have one
> PG replica on the 1TB OSD, then it would be possible to state "this
> pool cant write more than X G"
> but when it is full, there would be free space left on some of the
> 2TB-OSDs, but which the pool
> cant utilize. Probably same for uneven OSD hosts if you have those.
>
> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth
> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>
>> Dear Janne,
>>
>> yes, of course. But since we only have two pools here, this can not explain the difference.
>> The metadata is replicated (3 copies) across ssd drives, and we have < 3 TB of total raw storage for that.
>> So looking at the raw space usage, we can ignore that.
>>
>> All the rest is used for the ceph_data pool. So the ceph_data pool, in terms of raw storage, is about 50 % used.
>>
>> But in terms of storage shown for that pool, it's almost 63 % %USED.
>> So I guess this can purely be from bad balancing, correct?
>>
>> Cheers,
>> Oliver
>>
>> Am 20.10.18 um 19:49 schrieb Janne Johansson:
>>> Do mind that drives may have more than one pool on them, so RAW space
>>> is what it says, how much free space there is. Then the avail and
>>> %USED on per-pool stats will take replication into account, it can
>>> tell how much data you may write into that particular pool, given that
>>> pools replication or EC settings.
>>>
>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth
>>> <freyermuth@xxxxxxxxxxxxxxxxxx>:
>>>>
>>>> Dear Cephalopodians,
>>>>
>>>> as many others, I'm also a bit confused by "ceph df" output
>>>> in a pretty straightforward configuration.
>>>>
>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile.
>>>>
>>>> I get:
>>>> ----------------------------------------------------------------------------
>>>> # ceph df
>>>> GLOBAL:
>>>> SIZE AVAIL RAW USED %RAW USED
>>>> 824T 410T 414T 50.26
>>>> POOLS:
>>>> NAME ID USED %USED MAX AVAIL OBJECTS
>>>> cephfs_metadata 1 452M 0.05 860G 365774
>>>> cephfs_data 2 275T 62.68 164T 75056403
>>>> ----------------------------------------------------------------------------
>>>>
>>>> So about 50 % of raw space are used, but already ~63 % of filesystem space are used.
>>>> Is this purely from imperfect balancing?
>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % usage down to 37.12 %.
>>>>
>>>> We did not yet use the balancer plugin.
>>>> We don't have any pre-luminous clients.
>>>> In that setup, I take it that "upmap" mode would be recommended - correct?
>>>> Any "gotchas" using that on luminous?
>>>>
>>>> Cheers,
>>>> Oliver
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@xxxxxxxxxxxxxx
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>>
>>
>>
>
>
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
