Hi all,
ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)
We're having an issue with the spread of data across our OSDs. We have
108 OSDs in our cluster, all identical disk size, same number in each
server, and the same number of servers in each rack. So I'd hoped we'd
end up with a pretty balanced distribution of data across the disks.
However, the fullest is at 85% full and the most empty is at 40% full.
I've included the osd df output below, along with pool and crush rules.
I've also looked at the reweight-by-utilization command which would
apparently help:
# ceph osd test-reweight-by-utilization
moved 16 / 5715 (0.279965%)
avg 52.9167
stddev 7.20998 -> 7.15325 (expected baseline 7.24063)
min osd.45 with 31 -> 31 pgs (0.585827 -> 0.585827 * mean)
max osd.22 with 70 -> 68 pgs (1.32283 -> 1.28504 * mean)
oload 120
max_change 0.05
max_change_osds 4
average_utilization 0.6229
overload_utilization 0.7474
osd.22 weight 1.0000 -> 0.9500
osd.23 weight 1.0000 -> 0.9500
osd.53 weight 1.0000 -> 0.9500
osd.78 weight 1.0000 -> 0.9500
no change
But I'd like to make sure I understand why the problem is occuring
first so I can rule out a configuration issue, since it feels like the
cluster shouldn't be getting in to this state in the first place.
I have some suspicions that the number of PGs may be a bit low on some
pools, but autoscale-status is set to "on" or "warn" for every pool, so
it's happy with the current numbers. Does it play nice with CephFS?
Thanks for any advice.
Tim.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
22 hdd 3.63199 1.00000 3.6 TiB 3.1 TiB 3.1 TiB 450 MiB 7.6 GiB 557 GiB 85.04 1.37 70 up
23 hdd 3.63199 1.00000 3.6 TiB 2.9 TiB 2.9 TiB 459 MiB 7.5 GiB 759 GiB 79.64 1.28 64 up
53 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 703 MiB 8.0 GiB 823 GiB 77.91 1.25 66 up
78 hdd 3.63799 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 187 MiB 5.9 GiB 851 GiB 77.15 1.24 61 up
26 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 432 MiB 7.7 GiB 854 GiB 77.07 1.24 61 up
39 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 503 MiB 7.2 GiB 874 GiB 76.55 1.23 65 up
42 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.7 TiB 439 MiB 6.6 GiB 909 GiB 75.59 1.21 60 up
101 hdd 3.63820 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 306 MiB 7.1 GiB 913 GiB 75.50 1.21 61 up
87 hdd 3.63820 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 539 MiB 7.5 GiB 921 GiB 75.27 1.21 61 up
59 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 721 MiB 7.9 GiB 957 GiB 74.30 1.19 64 up
79 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 950 MiB 9.0 GiB 970 GiB 73.95 1.19 58 up
34 hdd 3.63199 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 202 MiB 6.0 GiB 974 GiB 73.85 1.19 57 up
60 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.6 TiB 668 MiB 7.2 GiB 1009 GiB 72.91 1.17 59 up
18 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 453 MiB 6.5 GiB 1021 GiB 72.59 1.17 60 up
74 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 693 MiB 7.5 GiB 1.0 TiB 72.12 1.16 62 up
19 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 655 MiB 7.9 GiB 1.0 TiB 71.71 1.15 63 up
69 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 445 MiB 6.2 GiB 1.0 TiB 71.70 1.15 65 up
43 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 170 MiB 4.7 GiB 1.0 TiB 71.62 1.15 63 up
97 hdd 3.63820 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 276 MiB 5.7 GiB 1.0 TiB 71.33 1.15 66 up
67 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 430 MiB 6.3 GiB 1.0 TiB 71.22 1.14 54 up
68 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 419 MiB 6.6 GiB 1.1 TiB 70.68 1.13 58 up
31 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.5 TiB 419 MiB 5.2 GiB 1.1 TiB 70.16 1.13 63 up
48 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.5 TiB 211 MiB 5.0 GiB 1.1 TiB 70.13 1.13 56 up
73 hdd 3.63799 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 765 MiB 7.1 GiB 1.1 TiB 69.52 1.12 57 up
98 hdd 3.63820 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 552 MiB 7.1 GiB 1.1 TiB 68.72 1.10 60 up
58 hdd 3.63799 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 427 MiB 6.3 GiB 1.2 TiB 68.39 1.10 53 up
14 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 409 MiB 6.0 GiB 1.2 TiB 68.06 1.09 65 up
47 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 166 MiB 5.5 GiB 1.2 TiB 67.84 1.09 55 up
9 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 419 MiB 5.9 GiB 1.2 TiB 67.78 1.09 58 up
90 hdd 3.63820 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 277 MiB 6.3 GiB 1.2 TiB 67.56 1.08 57 up
12 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.4 TiB 649 MiB 6.6 GiB 1.2 TiB 67.41 1.08 63 up
49 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 169 MiB 5.1 GiB 1.2 TiB 66.95 1.07 54 up
41 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 203 MiB 5.5 GiB 1.2 TiB 66.95 1.07 54 up
55 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 179 MiB 5.1 GiB 1.2 TiB 66.79 1.07 47 up
2 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 207 MiB 5.2 GiB 1.2 TiB 66.79 1.07 59 up
36 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 194 MiB 5.4 GiB 1.2 TiB 66.71 1.07 52 up
62 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 167 MiB 5.1 GiB 1.2 TiB 66.66 1.07 50 up
17 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 200 MiB 5.0 GiB 1.2 TiB 66.53 1.07 53 up
52 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 404 MiB 5.0 GiB 1.2 TiB 66.13 1.06 61 up
1 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 438 MiB 5.4 GiB 1.2 TiB 66.12 1.06 64 up
96 hdd 3.63820 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 15 MiB 4.2 GiB 1.2 TiB 65.92 1.06 53 up
54 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 666 MiB 6.4 GiB 1.3 TiB 65.47 1.05 63 up
37 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 412 MiB 6.6 GiB 1.3 TiB 64.87 1.04 54 up
93 hdd 3.63820 1.00000 3.6 TiB 2.4 TiB 2.3 TiB 304 MiB 6.0 GiB 1.3 TiB 64.63 1.04 57 up
15 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 200 MiB 5.8 GiB 1.3 TiB 64.22 1.03 51 up
66 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 427 MiB 5.2 GiB 1.3 TiB 63.98 1.03 50 up
28 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 169 MiB 4.6 GiB 1.3 TiB 63.75 1.02 54 up
30 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 400 MiB 5.6 GiB 1.3 TiB 63.28 1.02 53 up
81 hdd 3.63820 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 55 MiB 5.6 GiB 1.3 TiB 63.02 1.01 47 up
21 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 180 MiB 4.2 GiB 1.3 TiB 63.00 1.01 51 up
64 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 667 MiB 7.0 GiB 1.4 TiB 62.73 1.01 56 up
57 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 167 MiB 4.9 GiB 1.4 TiB 62.49 1.00 48 up
8 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 682 MiB 6.9 GiB 1.4 TiB 62.25 1.00 52 up
95 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 791 MiB 7.8 GiB 1.4 TiB 61.74 0.99 57 up
75 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 1.1 GiB 8.1 GiB 1.4 TiB 61.30 0.98 53 up
103 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 34 MiB 4.5 GiB 1.4 TiB 61.29 0.98 50 up
89 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 25 MiB 4.3 GiB 1.4 TiB 60.99 0.98 53 up
71 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 421 MiB 5.3 GiB 1.4 TiB 60.95 0.98 56 up
5 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 158 MiB 3.8 GiB 1.4 TiB 60.69 0.97 50 up
61 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 166 MiB 4.6 GiB 1.4 TiB 60.29 0.97 55 up
94 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 302 MiB 6.1 GiB 1.4 TiB 60.26 0.97 55 up
72 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 170 MiB 4.0 GiB 1.5 TiB 59.91 0.96 51 up
91 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 18 MiB 3.9 GiB 1.5 TiB 59.75 0.96 51 up
38 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 429 MiB 5.4 GiB 1.5 TiB 59.42 0.95 49 up
33 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 980 MiB 6.9 GiB 1.5 TiB 59.36 0.95 52 up
3 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 174 MiB 4.5 GiB 1.5 TiB 58.79 0.94 49 up
13 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 177 MiB 3.9 GiB 1.5 TiB 58.75 0.94 54 up
0 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 179 MiB 4.7 GiB 1.5 TiB 58.63 0.94 50 up
77 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 429 MiB 5.3 GiB 1.5 TiB 58.59 0.94 50 up
46 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 406 MiB 5.8 GiB 1.5 TiB 58.25 0.94 49 up
84 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 849 MiB 7.1 GiB 1.5 TiB 57.97 0.93 52 up
63 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 144 MiB 4.3 GiB 1.5 TiB 57.94 0.93 55 up
25 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 153 MiB 4.8 GiB 1.5 TiB 57.87 0.93 53 up
4 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 682 MiB 6.4 GiB 1.5 TiB 57.76 0.93 47 up
102 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 49 MiB 4.9 GiB 1.6 TiB 57.25 0.92 45 up
10 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 446 MiB 4.5 GiB 1.6 TiB 57.24 0.92 53 up
88 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 550 MiB 6.6 GiB 1.6 TiB 56.57 0.91 50 up
92 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 39 MiB 4.5 GiB 1.6 TiB 56.53 0.91 44 up
56 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.0 TiB 156 MiB 4.0 GiB 1.6 TiB 56.40 0.91 54 up
85 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.0 TiB 35 MiB 4.3 GiB 1.6 TiB 56.38 0.91 47 up
40 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 156 MiB 3.9 GiB 1.6 TiB 56.22 0.90 47 up
76 hdd 3.63799 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 166 MiB 4.6 GiB 1.6 TiB 56.18 0.90 46 up
83 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 35 MiB 4.3 GiB 1.6 TiB 56.17 0.90 49 up
50 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 419 MiB 5.2 GiB 1.6 TiB 55.29 0.89 51 up
35 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 415 MiB 5.5 GiB 1.6 TiB 55.27 0.89 45 up
106 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 266 MiB 4.8 GiB 1.6 TiB 55.19 0.89 47 up
44 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 378 MiB 5.2 GiB 1.6 TiB 55.11 0.88 45 up
29 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 416 MiB 4.8 GiB 1.6 TiB 54.87 0.88 52 up
86 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 559 MiB 6.7 GiB 1.6 TiB 54.80 0.88 47 up
82 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 298 MiB 5.2 GiB 1.6 TiB 54.67 0.88 49 up
65 hdd 3.63799 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 918 MiB 6.7 GiB 1.7 TiB 54.15 0.87 51 up
70 hdd 3.63799 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 675 MiB 5.7 GiB 1.7 TiB 53.49 0.86 50 up
11 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 165 MiB 4.6 GiB 1.7 TiB 52.97 0.85 46 up
24 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 156 MiB 4.2 GiB 1.7 TiB 52.80 0.85 44 up
105 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 38 MiB 4.4 GiB 1.7 TiB 52.45 0.84 47 up
104 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 288 MiB 5.5 GiB 1.7 TiB 52.09 0.84 38 up
107 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 290 MiB 5.5 GiB 1.8 TiB 51.57 0.83 41 up
20 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 381 MiB 5.2 GiB 1.8 TiB 51.48 0.83 51 up
32 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 116 MiB 4.1 GiB 1.8 TiB 50.37 0.81 48 up
100 hdd 3.63820 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 539 MiB 6.1 GiB 1.8 TiB 50.21 0.81 47 up
7 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 173 MiB 3.9 GiB 1.8 TiB 50.14 0.81 45 up
51 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 640 MiB 5.9 GiB 1.8 TiB 49.97 0.80 46 up
16 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 158 MiB 3.9 GiB 1.8 TiB 49.87 0.80 41 up
99 hdd 3.63820 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 258 MiB 4.4 GiB 1.8 TiB 49.81 0.80 44 up
6 hdd 3.63199 1.00000 3.6 TiB 1.7 TiB 1.7 TiB 149 MiB 3.8 GiB 1.9 TiB 47.96 0.77 42 up
80 hdd 3.63799 1.00000 3.6 TiB 1.7 TiB 1.7 TiB 640 MiB 5.1 GiB 1.9 TiB 46.48 0.75 37 up
27 hdd 3.63199 1.00000 3.6 TiB 1.6 TiB 1.6 TiB 338 MiB 3.9 GiB 2.1 TiB 42.72 0.69 36 up
45 hdd 3.63199 1.00000 3.6 TiB 1.5 TiB 1.5 TiB 150 MiB 3.2 GiB 2.2 TiB 40.28 0.65 31 up
TOTAL 393 TiB 245 TiB 244 TiB 38 GiB 605 GiB 148 TiB 62.28
MIN/MAX VAR: 0.65/1.37 STDDEV: 8.55
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 393 TiB 148 TiB 245 TiB 245 TiB 62.29
TOTAL 393 TiB 148 TiB 245 TiB 245 TiB 62.29
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
pool28 28 256 9.9 TiB 2.61M 30 TiB 43.28 13 TiB
pool29 29 256 9.5 TiB 2.48M 28 TiB 42.13 13 TiB
pool36 36 128 6.0 TiB 1.58M 18 TiB 31.67 13 TiB
pool39 39 256 20 TiB 5.20M 60 TiB 60.37 13 TiB
pool43 43 32 1.9 TiB 503.92k 5.7 TiB 12.79 13 TiB
pool44 44 16 385 GiB 133.10k 1.1 TiB 2.80 13 TiB
pool45 45 32 9.1 KiB 0 27 KiB 0 13 TiB
pool46 46 32 1.3 TiB 236.34k 3.9 TiB 9.14 13 TiB
pool47 47 128 4.0 TiB 1.04M 12 TiB 23.35 13 TiB
pool48 48 32 6.0 MiB 0 18 MiB 0 13 TiB
pool49 49 32 8.3 GiB 1.49M 25 GiB 0.06 13 TiB
pool50 50 32 3.0 MiB 3 9.2 MiB 0 13 TiB
pool51 51 128 32 TiB 32.47M 55 TiB 58.30 26 TiB
pool53 52 1 56 KiB 108 169 KiB 0 13 TiB
pool53 53 128 3.3 TiB 864.88k 9.9 TiB 20.21 13 TiB
pool54 54 32 2.2 MiB 4 7.0 MiB 0 13 TiB
pool57 57 128 14 TiB 3.55M 21 TiB 34.80 26 TiB
pool 28 'pool28' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301 flags hashpspool,nearfull stripe_width 0 application rbd
pool 29 'pool29' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301 flags hashpspool,nearfull stripe_width 0 application rbd
pool 36 'pool36' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 150301 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
pool 39 'pool39' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
pool 43 'pool43' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor 0/140049/140047 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
pool 44 'pool44' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 150301 lfor 0/66919/66917 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
pool 45 'pool45' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor 0/58489/67317 flags hashpspool,nearfull stripe_width 0 application rbd
pool 46 'pool46' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor 0/66943/66941 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
pool 47 'pool47' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 150301 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
pool 48 'pool48' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor 0/64382/67327 flags hashpspool,nearfull stripe_width 0 application cephfs
pool 49 'pool49' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor 0/66981/67337 flags hashpspool,nearfull stripe_width 0 application cephfs
pool 50 'pool50' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor 0/66733/67322 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application benchmarks
pool 51 'pool51' erasure profile ec42 size 6 min_size 4 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 150301 lfor 0/80117/81361 flags hashpspool,ec_overwrites,nearfull,selfmanaged_snaps stripe_width 16384 application cephfs
pool 52 'pool52' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode warn last_change 150301 flags hashpspool,nearfull stripe_width 0 pg_num_min 1 application mgr_devicehealth
pool 53 'pool53' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 150301 lfor 0/92746/101633 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
pool 54 'pool54' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 150301 lfor 0/96348/96346 flags hashpspool,nearfull stripe_width 0 application rbd
pool 57 'pool57' erasure profile ec42 size 6 min_size 4 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 150301 lfor 0/96618/101635 flags hashpspool,ec_overwrites,nearfull,selfmanaged_snaps stripe_width 16384 application rbd
[
{
"rule_id": 0,
"rule_name": "replicated_ruleset",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "rack"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "ec42_ruleset",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "choose_indep",
"num": 3,
"type": "rack"
},
{
"op": "chooseleaf_indep",
"num": 2,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "erasure-code",
"ruleset": 2,
"type": 3,
"min_size": 3,
"max_size": 3,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 392.58435 root default
-8 130.86145 rack R1
-4 43.62048 host C1
1 hdd 3.63199 osd.1 up 1.00000 1.00000
12 hdd 3.63199 osd.12 up 1.00000 1.00000
16 hdd 3.63199 osd.16 up 1.00000 1.00000
22 hdd 3.63199 osd.22 up 1.00000 1.00000
27 hdd 3.63199 osd.27 up 1.00000 1.00000
31 hdd 3.63199 osd.31 up 1.00000 1.00000
55 hdd 3.63799 osd.55 up 1.00000 1.00000
57 hdd 3.63799 osd.57 up 1.00000 1.00000
58 hdd 3.63799 osd.58 up 1.00000 1.00000
81 hdd 3.63820 osd.81 up 1.00000 1.00000
82 hdd 3.63820 osd.82 up 1.00000 1.00000
99 hdd 3.63820 osd.99 up 1.00000 1.00000
-12 43.62048 host C2
37 hdd 3.63199 osd.37 up 1.00000 1.00000
40 hdd 3.63199 osd.40 up 1.00000 1.00000
44 hdd 3.63199 osd.44 up 1.00000 1.00000
47 hdd 3.63199 osd.47 up 1.00000 1.00000
50 hdd 3.63199 osd.50 up 1.00000 1.00000
53 hdd 3.63199 osd.53 up 1.00000 1.00000
56 hdd 3.63799 osd.56 up 1.00000 1.00000
59 hdd 3.63799 osd.59 up 1.00000 1.00000
60 hdd 3.63799 osd.60 up 1.00000 1.00000
83 hdd 3.63820 osd.83 up 1.00000 1.00000
84 hdd 3.63820 osd.84 up 1.00000 1.00000
100 hdd 3.63820 osd.100 up 1.00000 1.00000
-2 43.62048 host C3
0 hdd 3.63199 osd.0 up 1.00000 1.00000
2 hdd 3.63199 osd.2 up 1.00000 1.00000
3 hdd 3.63199 osd.3 up 1.00000 1.00000
4 hdd 3.63199 osd.4 up 1.00000 1.00000
5 hdd 3.63199 osd.5 up 1.00000 1.00000
6 hdd 3.63199 osd.6 up 1.00000 1.00000
54 hdd 3.63799 osd.54 up 1.00000 1.00000
61 hdd 3.63799 osd.61 up 1.00000 1.00000
62 hdd 3.63799 osd.62 up 1.00000 1.00000
87 hdd 3.63820 osd.87 up 1.00000 1.00000
88 hdd 3.63820 osd.88 up 1.00000 1.00000
101 hdd 3.63820 osd.101 up 1.00000 1.00000
-9 130.86145 rack R2
-3 43.62048 host C4
13 hdd 3.63199 osd.13 up 1.00000 1.00000
20 hdd 3.63199 osd.20 up 1.00000 1.00000
23 hdd 3.63199 osd.23 up 1.00000 1.00000
28 hdd 3.63199 osd.28 up 1.00000 1.00000
33 hdd 3.63199 osd.33 up 1.00000 1.00000
35 hdd 3.63199 osd.35 up 1.00000 1.00000
63 hdd 3.63799 osd.63 up 1.00000 1.00000
65 hdd 3.63799 osd.65 up 1.00000 1.00000
67 hdd 3.63799 osd.67 up 1.00000 1.00000
93 hdd 3.63820 osd.93 up 1.00000 1.00000
94 hdd 3.63820 osd.94 up 1.00000 1.00000
103 hdd 3.63820 osd.103 up 1.00000 1.00000
-11 43.62048 host C5
36 hdd 3.63199 osd.36 up 1.00000 1.00000
39 hdd 3.63199 osd.39 up 1.00000 1.00000
42 hdd 3.63199 osd.42 up 1.00000 1.00000
45 hdd 3.63199 osd.45 up 1.00000 1.00000
48 hdd 3.63199 osd.48 up 1.00000 1.00000
51 hdd 3.63199 osd.51 up 1.00000 1.00000
69 hdd 3.63799 osd.69 up 1.00000 1.00000
70 hdd 3.63799 osd.70 up 1.00000 1.00000
71 hdd 3.63799 osd.71 up 1.00000 1.00000
95 hdd 3.63820 osd.95 up 1.00000 1.00000
96 hdd 3.63820 osd.96 up 1.00000 1.00000
104 hdd 3.63820 osd.104 up 1.00000 1.00000
-7 43.62048 host C6
10 hdd 3.63199 osd.10 up 1.00000 1.00000
15 hdd 3.63199 osd.15 up 1.00000 1.00000
19 hdd 3.63199 osd.19 up 1.00000 1.00000
25 hdd 3.63199 osd.25 up 1.00000 1.00000
30 hdd 3.63199 osd.30 up 1.00000 1.00000
34 hdd 3.63199 osd.34 up 1.00000 1.00000
64 hdd 3.63799 osd.64 up 1.00000 1.00000
66 hdd 3.63799 osd.66 up 1.00000 1.00000
68 hdd 3.63799 osd.68 up 1.00000 1.00000
91 hdd 3.63820 osd.91 up 1.00000 1.00000
92 hdd 3.63820 osd.92 up 1.00000 1.00000
102 hdd 3.63820 osd.102 up 1.00000 1.00000
-10 130.86145 rack R3
-5 43.62048 host C7
9 hdd 3.63199 osd.9 up 1.00000 1.00000
14 hdd 3.63199 osd.14 up 1.00000 1.00000
17 hdd 3.63199 osd.17 up 1.00000 1.00000
24 hdd 3.63199 osd.24 up 1.00000 1.00000
29 hdd 3.63199 osd.29 up 1.00000 1.00000
32 hdd 3.63199 osd.32 up 1.00000 1.00000
73 hdd 3.63799 osd.73 up 1.00000 1.00000
76 hdd 3.63799 osd.76 up 1.00000 1.00000
79 hdd 3.63799 osd.79 up 1.00000 1.00000
97 hdd 3.63820 osd.97 up 1.00000 1.00000
98 hdd 3.63820 osd.98 up 1.00000 1.00000
105 hdd 3.63820 osd.105 up 1.00000 1.00000
-13 43.62048 host C8
38 hdd 3.63199 osd.38 up 1.00000 1.00000
41 hdd 3.63199 osd.41 up 1.00000 1.00000
43 hdd 3.63199 osd.43 up 1.00000 1.00000
46 hdd 3.63199 osd.46 up 1.00000 1.00000
49 hdd 3.63199 osd.49 up 1.00000 1.00000
52 hdd 3.63199 osd.52 up 1.00000 1.00000
72 hdd 3.63799 osd.72 up 1.00000 1.00000
75 hdd 3.63799 osd.75 up 1.00000 1.00000
78 hdd 3.63799 osd.78 up 1.00000 1.00000
85 hdd 3.63820 osd.85 up 1.00000 1.00000
86 hdd 3.63820 osd.86 up 1.00000 1.00000
107 hdd 3.63820 osd.107 up 1.00000 1.00000
-6 43.62048 host C9
7 hdd 3.63199 osd.7 up 1.00000 1.00000
8 hdd 3.63199 osd.8 up 1.00000 1.00000
11 hdd 3.63199 osd.11 up 1.00000 1.00000
18 hdd 3.63199 osd.18 up 1.00000 1.00000
21 hdd 3.63199 osd.21 up 1.00000 1.00000
26 hdd 3.63199 osd.26 up 1.00000 1.00000
74 hdd 3.63799 osd.74 up 1.00000 1.00000
77 hdd 3.63799 osd.77 up 1.00000 1.00000
80 hdd 3.63799 osd.80 up 1.00000 1.00000
89 hdd 3.63820 osd.89 up 1.00000 1.00000
90 hdd 3.63820 osd.90 up 1.00000 1.00000
106 hdd 3.63820 osd.106 up 1.00000 1.00000
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
