What does your crush map look like? Also a `ceph df` output. You're optimizing your map for pool #5, if there are other pools with a significant amount of data, then your going to be off on your cluster balance.
A big question for balancing a cluster is how big are your PGs? If your primary data pool has PGs that are 100GB in size, then even if you balanced the cluster such that all of the osds were within 1 PG of each other (assuming your average OSD size <1TB), then your OSDs would be 10% apart in disk usage from each other and 100GB.
On Thu, Oct 26, 2017 at 4:44 AM Stefan Priebe - Profihost AG <s.priebe@xxxxxxxxxxxx> wrote:
Hello,
while trying to optimize a ceph cluster running jewel i get the
following output:
2017-10-26 10:43:27,615 argv = optimize --crushmap
/home/spriebe/ceph.report --out-path /home/spriebe/optimized.crush
--pool 5 --pool=5 --choose-args=5 --replication-count=3 --pg-num=4096
--pgp-num=4096 --rule=data --out-version=j --no-positions
2017-10-26 10:43:27,646 root optimizing
2017-10-26 10:43:29,329 root already optimized
2017-10-26 10:43:29,337 cloud1-1475 optimizing
2017-10-26 10:43:29,348 cloud1-1474 optimizing
2017-10-26 10:43:29,353 cloud1-1473 optimizing
2017-10-26 10:43:29,361 cloud1-1467 optimizing
2017-10-26 10:43:30,118 cloud1-1473 already optimized
2017-10-26 10:43:30,126 cloud1-1472 optimizing
2017-10-26 10:43:30,177 cloud1-1474 already optimized
2017-10-26 10:43:30,178 cloud1-1467 already optimized
2017-10-26 10:43:30,185 cloud1-1471 optimizing
2017-10-26 10:43:30,193 cloud1-1470 optimizing
2017-10-26 10:43:30,301 cloud1-1475 already optimized
2017-10-26 10:43:30,310 cloud1-1469 optimizing
2017-10-26 10:43:30,855 cloud1-1472 already optimized
2017-10-26 10:43:30,864 cloud1-1468 optimizing
2017-10-26 10:43:31,020 cloud1-1470 already optimized
2017-10-26 10:43:31,075 cloud1-1471 already optimized
2017-10-26 10:43:31,079 cloud1-1469 already optimized
2017-10-26 10:43:31,460 cloud1-1468 already optimized
But this one is heavily inbalanced if you look at the AVAIL GB.
ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
31 1.00000 1.00000 931G 756G 174G 81.24 1.24 204
32 1.00000 1.00000 931G 750G 180G 80.58 1.23 202
33 1.00000 1.00000 931G 774G 157G 83.13 1.27 209
34 1.00000 1.00000 931G 743G 187G 79.89 1.22 200
48 1.00000 1.00000 931G 780G 150G 83.79 1.28 210
49 1.00000 1.00000 931G 778G 152G 83.64 1.27 209
50 0.79999 1.00000 819G 631G 188G 77.00 1.17 170
21 1.00000 0.95001 931G 729G 201G 78.32 1.19 197
22 1.00000 1.00000 931G 756G 174G 81.28 1.24 204
23 1.00000 1.00000 931G 778G 152G 83.57 1.27 210
24 1.00000 0.95001 931G 760G 171G 81.63 1.24 205
46 1.00000 1.00000 931G 756G 174G 81.30 1.24 203
47 1.00000 1.00000 931G 768G 162G 82.57 1.26 207
55 0.79999 1.00000 819G 602G 217G 73.46 1.12 162
11 1.00000 1.00000 931G 628G 302G 67.47 1.03 169
12 1.00000 1.00000 931G 630G 300G 67.68 1.03 170
13 1.00000 1.00000 931G 629G 301G 67.63 1.03 170
14 1.00000 1.00000 931G 644G 286G 69.20 1.05 173
45 1.00000 1.00000 931G 635G 295G 68.23 1.04 171
56 0.79999 1.00000 819G 509G 309G 62.18 0.95 137
40 1.72670 1.00000 1768G 774G 994G 43.78 0.67 209
51 1.00000 0.95001 931G 784G 146G 84.26 1.28 212
52 1.00000 1.00000 931G 768G 162G 82.52 1.26 207
53 1.00000 1.00000 931G 752G 178G 80.78 1.23 203
54 1.00000 1.00000 931G 750G 180G 80.58 1.23 202
38 1.00000 1.00000 931G 768G 162G 82.58 1.26 207
39 1.00000 1.00000 931G 768G 162G 82.51 1.26 207
57 0.79999 1.00000 819G 616G 203G 75.17 1.15 167
43 1.00000 1.00000 931G 636G 294G 68.41 1.04 171
44 1.00000 1.00000 931G 630G 300G 67.70 1.03 170
36 1.00000 1.00000 931G 626G 304G 67.33 1.03 170
37 1.00000 1.00000 931G 637G 293G 68.43 1.04 171
58 0.79999 1.00000 819G 508G 311G 62.03 0.95 137
41 0.87000 1.00000 888G 549G 339G 61.79 0.94 148
42 1.72670 1.00000 1768G 382G 1385G 21.65 0.33 103
65 1.72670 1.00000 1768G 383G 1384G 21.69 0.33 103
1 1.00000 1.00000 931G 760G 170G 81.70 1.24 205
2 1.00000 1.00000 931G 768G 162G 82.57 1.26 207
3 1.00000 0.95001 931G 732G 198G 78.71 1.20 197
30 1.00000 1.00000 931G 771G 159G 82.84 1.26 209
35 1.00000 1.00000 931G 751G 179G 80.70 1.23 202
59 0.79999 1.00000 819G 599G 220G 73.08 1.11 162
0 0.84999 1.00000 874G 662G 211G 75.81 1.16 179
4 1.00000 1.00000 931G 633G 297G 68.07 1.04 171
5 1.00000 1.00000 931G 632G 298G 67.94 1.04 171
60 0.79999 1.00000 819G 509G 310G 62.12 0.95 137
29 0.87000 1.00000 888G 553G 335G 62.29 0.95 149
7 0.87000 1.00000 888G 554G 333G 62.44 0.95 149
28 0.87000 1.00000 888G 551G 336G 62.09 0.95 149
64 1.73000 1.00000 1768G 1088G 679G 61.57 0.94 293
6 1.72670 1.00000 1768G 472G 1295G 26.71 0.41 127
66 1.72670 1.00000 1768G 475G 1292G 26.88 0.41 128
8 1.00000 1.00000 931G 625G 305G 67.14 1.02 169
9 1.00000 1.00000 931G 631G 299G 67.82 1.03 170
61 0.79999 1.00000 819G 507G 312G 61.90 0.94 137
15 0.87000 1.00000 888G 550G 338G 61.89 0.94 148
26 0.87000 1.00000 888G 551G 337G 62.04 0.95 149
27 0.84999 1.00000 874G 541G 333G 61.90 0.94 146
63 1.73000 1.00000 1768G 1101G 666G 62.31 0.95 297
10 1.72670 1.00000 1768G 483G 1284G 27.34 0.42 131
67 1.72670 1.00000 1768G 482G 1285G 27.31 0.42 130
16 1.00000 1.00000 931G 786G 144G 84.46 1.29 212
17 1.00000 1.00000 931G 783G 147G 84.12 1.28 211
18 1.00000 1.00000 931G 747G 183G 80.28 1.22 201
19 1.00000 1.00000 931G 758G 172G 81.50 1.24 205
20 1.00000 1.00000 931G 759G 171G 81.59 1.24 205
25 1.00000 1.00000 931G 766G 164G 82.35 1.25 207
62 0.79999 1.00000 819G 620G 199G 75.68 1.15 166
31 1.00000 1.00000 931G 756G 174G 81.24 1.24 204
32 1.01900 1.00000 931G 750G 180G 80.58 1.23 202
33 1.00000 1.00000 931G 774G 157G 83.13 1.27 209
34 0.92699 1.00000 931G 743G 187G 79.89 1.22 200
48 1.00000 1.00000 931G 780G 150G 83.79 1.28 210
49 1.02399 1.00000 931G 778G 152G 83.64 1.27 209
50 0.82999 1.00000 819G 631G 188G 77.00 1.17 170
21 1.12793 0.95001 931G 729G 201G 78.32 1.19 197
22 0.93199 1.00000 931G 756G 174G 81.28 1.24 204
23 0.94606 1.00000 931G 778G 152G 83.57 1.27 210
24 0.95599 0.95001 931G 760G 171G 81.63 1.24 205
46 1.01500 1.00000 931G 756G 174G 81.30 1.24 203
47 0.99500 1.00000 931G 768G 162G 82.57 1.26 207
55 0.82899 1.00000 819G 602G 217G 73.46 1.12 162
11 0.78575 1.00000 931G 628G 302G 67.47 1.03 169
12 0.89369 1.00000 931G 630G 300G 67.68 1.03 170
13 0.84279 1.00000 931G 629G 301G 67.63 1.03 170
14 0.77834 1.00000 931G 644G 286G 69.20 1.05 173
45 0.83331 1.00000 931G 635G 295G 68.23 1.04 171
56 0.74358 1.00000 819G 509G 309G 62.18 0.95 137
40 0.96149 1.00000 1768G 774G 994G 43.78 0.67 209
51 1.03899 0.95001 931G 784G 146G 84.26 1.28 212
52 0.98000 1.00000 931G 768G 162G 82.52 1.26 207
53 1.01900 1.00000 931G 752G 178G 80.78 1.23 203
54 0.98000 1.00000 931G 750G 180G 80.58 1.23 202
38 0.94199 1.00000 931G 768G 162G 82.58 1.26 207
39 1.03899 1.00000 931G 768G 162G 82.51 1.26 207
57 0.79999 1.00000 819G 616G 203G 75.17 1.15 167
43 0.77654 1.00000 931G 636G 294G 68.41 1.04 171
44 0.85941 1.00000 931G 630G 300G 67.70 1.03 170
36 0.85104 1.00000 931G 626G 304G 67.33 1.03 170
37 0.80946 1.00000 931G 637G 293G 68.43 1.04 171
58 0.67404 1.00000 819G 508G 311G 62.03 0.95 137
41 0.68245 1.00000 888G 549G 339G 61.79 0.94 148
42 0.49387 1.00000 1768G 382G 1385G 21.65 0.33 103
65 0.52313 1.00000 1768G 383G 1384G 21.69 0.33 103
1 0.96100 1.00000 931G 760G 170G 81.70 1.24 205
2 0.94199 1.00000 931G 768G 162G 82.57 1.26 207
3 1.03499 0.95001 931G 732G 198G 78.71 1.20 197
30 1.00000 1.00000 931G 771G 159G 82.84 1.26 209
35 1.07399 1.00000 931G 751G 179G 80.70 1.23 202
59 0.87000 1.00000 819G 599G 220G 73.08 1.11 162
0 0.76900 1.00000 874G 662G 211G 75.81 1.16 179
4 0.73238 1.00000 931G 633G 297G 68.07 1.04 171
5 0.88098 1.00000 931G 632G 298G 67.94 1.04 171
60 0.67899 1.00000 819G 509G 310G 62.12 0.95 137
29 0.72182 1.00000 888G 553G 335G 62.29 0.95 149
7 0.71684 1.00000 888G 554G 333G 62.44 0.95 149
28 0.64395 1.00000 888G 551G 336G 62.09 0.95 149
64 1.46759 1.00000 1768G 1088G 679G 61.57 0.94 293
6 0.59332 1.00000 1768G 472G 1295G 26.71 0.41 127
66 0.68506 1.00000 1768G 475G 1292G 26.88 0.41 128
8 0.89154 1.00000 931G 625G 305G 67.14 1.02 169
9 0.83815 1.00000 931G 631G 299G 67.82 1.03 170
61 0.64093 1.00000 819G 507G 312G 61.90 0.94 137
15 0.74059 1.00000 888G 550G 338G 61.89 0.94 148
26 0.65033 1.00000 888G 551G 337G 62.04 0.95 149
27 0.61040 1.00000 874G 541G 333G 61.90 0.94 146
63 1.44260 1.00000 1768G 1101G 666G 62.31 0.95 297
10 0.65889 1.00000 1768G 483G 1284G 27.34 0.42 131
67 0.60854 1.00000 1768G 482G 1285G 27.31 0.42 130
16 0.98000 1.00000 931G 786G 144G 84.46 1.29 212
17 1.11299 1.00000 931G 783G 147G 84.12 1.28 211
18 1.12399 1.00000 931G 747G 183G 80.28 1.22 201
19 0.87099 1.00000 931G 758G 172G 81.50 1.24 205
20 0.95099 1.00000 931G 759G 171G 81.59 1.24 205
25 1.04700 1.00000 931G 766G 164G 82.35 1.25 207
62 0.71399 1.00000 819G 620G 199G 75.68 1.15 166
TOTAL 69475G 45596G 23878G 65.63
MIN/MAX VAR: 0.33/1.29 STDDEV: 16.64
crush tool is 1.0.35
Greets,
Stefan
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
