Hi,
> On Sep 18, 2017, at 9:51 AM, Florian Haas <florian@xxxxxxxxxxx> wrote:
>
> For Josh's and others' benefit, I think you might want to share how
> many nodes you operate, as that would be quite relevant to the
> discussion.
Sure. See the OSD tree at the end.
We’re doing the typical SSD/non-SSD pool separation. Currently we effectively only use 2 pools: rbd.hdd and rbd.ssd. The ~4TB OSDs in the rbd.hdd pool are “capacity endurance” SSDs (Micron S610DC). We have 10 machines at the moment with 10 OSDs on average (2 SSD, 1-2 capacity SSD and 6-7 HDDs).
cluster d4b91002-eaf4-11e2-bc7c-0200000311c1
health HEALTH_OK
monmap e43: 5 mons at {cartman06=172.22.4.42:6789/0,cartman07=172.22.4.43:6789/0,cartman11=172.22.4.54:6789/0,cartman15=172.22.4.69:6789/0,cartman16=172.22.4.70:6789/0}
election epoch 7278, quorum 0,1,2,3,4 cartman06,cartman07,cartman11,cartman15,cartman16
osdmap e1360049: 96 osds: 95 up, 95 in
pgmap v115515846: 6336 pgs, 5 pools, 28786 GB data, 8244 kobjects
92181 GB used, 69333 GB / 157 TB avail
6335 active+clean
1 active+clean+scrubbing+deep
client io 379 MB/s rd, 197 MB/s wr, 10313 op/s
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-11 20.00000 root ssd
-10 20.00000 datacenter rzob-ssd
-9 6.00000 rack OB-1.R206.A-5-ssd
-7 2.00000 host cartman06-ssd
13 1.00000 osd.13 up 1.00000 1.00000
87 1.00000 osd.87 up 0.89999 1.00000
-20 2.00000 host cartman09-ssd
47 1.00000 osd.47 up 1.00000 1.00000
24 1.00000 osd.24 up 1.00000 1.00000
-12 2.00000 host cartman11-ssd
43 1.00000 osd.43 up 1.00000 1.00000
57 1.00000 osd.57 up 1.00000 1.00000
-16 6.00000 rack OB-1.R206.D-5-ssd
-17 2.00000 host cartman07-ssd
44 1.00000 osd.44 up 0.95000 1.00000
56 1.00000 osd.56 up 1.00000 1.00000
-18 2.00000 host cartman08-ssd
45 1.00000 osd.45 up 1.00000 1.00000
6 1.00000 osd.6 up 1.00000 1.00000
-19 2.00000 host cartman10-ssd
46 1.00000 osd.46 up 1.00000 1.00000
27 1.00000 osd.27 up 1.00000 1.00000
-27 8.00000 rack OB-1.R206.A-6-ssd
-26 2.00000 host cartman16-ssd
75 1.00000 osd.75 up 1.00000 1.00000
71 1.00000 osd.71 up 1.00000 1.00000
-28 2.00000 host cartman15-ssd
83 1.00000 osd.83 up 1.00000 1.00000
80 1.00000 osd.80 up 1.00000 1.00000
-29 2.00000 host cartman18-ssd
62 1.00000 osd.62 up 1.00000 1.00000
69 1.00000 osd.69 up 1.00000 1.00000
-30 2.00000 host cartman17-ssd
50 1.00000 osd.50 up 1.00000 0.50000
54 1.00000 osd.54 up 1.00000 1.00000
-1 143.81467 root default
-4 143.81467 datacenter rzob
-3 34.28958 rack OB-1.R206.A-5
-8 12.58118 host cartman06
0 1.81799 osd.0 up 1.00000 0
3 1.81799 osd.3 up 1.00000 0
14 1.81799 osd.14 up 1.00000 0
37 1.81799 osd.37 up 1.00000 0
38 1.81799 osd.38 up 1.00000 0
86 3.49121 osd.86 up 1.00000 1.00000
-15 14.39722 host cartman09
21 1.81799 osd.21 up 1.00000 0
2 1.81799 osd.2 up 1.00000 0
25 1.81799 osd.25 up 1.00000 0
39 1.81799 osd.39 up 1.00000 0
40 1.81799 osd.40 up 1.00000 0
41 1.81799 osd.41 up 1.00000 0
60 3.48926 osd.60 up 1.00000 1.00000
-5 7.31119 host cartman11
5 0.54599 osd.5 up 1.00000 0.00999
26 0.54599 osd.26 up 1.00000 0.00999
8 0.54599 osd.8 up 1.00000 0.00999
9 0.54599 osd.9 up 1.00000 0.00999
11 0.54599 osd.11 up 1.00000 0.00999
12 0.54599 osd.12 up 1.00000 0.00999
23 0.54599 osd.23 up 1.00000 0.00999
30 3.48926 osd.30 up 1.00000 1.00000
-6 37.92352 rack OB-1.R206.D-5
-13 16.21521 host cartman07
1 1.81799 osd.1 up 1.00000 0
15 1.81799 osd.15 up 1.00000 0
16 1.81799 osd.16 up 1.00000 0
19 1.81799 osd.19 up 1.00000 0
33 1.81799 osd.33 up 1.00000 0
34 1.81799 osd.34 up 1.00000 0
35 1.81799 osd.35 up 1.00000 0
48 3.48926 osd.48 up 1.00000 1.00000
-14 14.39722 host cartman08
17 1.81799 osd.17 up 1.00000 0
18 1.81799 osd.18 up 1.00000 0
22 1.81799 osd.22 up 1.00000 0
31 1.81799 osd.31 up 1.00000 0
32 1.81799 osd.32 up 1.00000 0
36 1.81799 osd.36 up 1.00000 0
58 3.48926 osd.58 up 1.00000 1.00000
-2 7.31110 host cartman10
4 0.54599 osd.4 down 0 0.00999
7 0.54599 osd.7 up 1.00000 0.00999
10 0.54599 osd.10 up 1.00000 0.00999
20 0.54599 osd.20 up 1.00000 0.00999
28 0.54599 osd.28 up 1.00000 0.00999
29 0.54599 osd.29 up 1.00000 0.00999
55 0.54590 osd.55 up 1.00000 0.00999
49 3.48926 osd.49 up 1.00000 1.00000
-22 71.60156 rack OB-1.R206.A-6
-21 17.90039 host cartman17
42 1.81934 osd.42 up 1.00000 0
51 1.81934 osd.51 up 1.00000 0
52 1.81934 osd.52 up 1.00000 0
53 1.81934 osd.53 up 1.00000 0
59 1.81934 osd.59 up 1.00000 0
61 1.81934 osd.61 up 1.00000 0
91 3.49219 osd.91 up 1.00000 1.00000
93 3.49219 osd.93 up 1.00000 1.00000
-23 17.90039 host cartman18
63 1.81934 osd.63 up 1.00000 0
64 1.81934 osd.64 up 1.00000 0
65 1.81934 osd.65 up 1.00000 0
66 1.81934 osd.66 up 1.00000 0
67 1.81934 osd.67 up 1.00000 0
68 1.81934 osd.68 up 1.00000 0
94 3.49219 osd.94 up 1.00000 1.00000
95 3.49219 osd.95 up 1.00000 1.00000
-24 17.90039 host cartman16
70 1.81934 osd.70 up 1.00000 0
72 1.81934 osd.72 up 1.00000 0
73 1.81934 osd.73 up 1.00000 0
74 1.81934 osd.74 up 1.00000 0
76 1.81934 osd.76 up 1.00000 0
77 1.81934 osd.77 up 1.00000 0
88 3.49219 osd.88 up 1.00000 1.00000
89 3.49219 osd.89 up 1.00000 1.00000
-25 17.90039 host cartman15
78 1.81934 osd.78 up 1.00000 0
79 1.81934 osd.79 up 1.00000 0
81 1.81934 osd.81 up 1.00000 0
82 1.81934 osd.82 up 1.00000 0
84 1.81934 osd.84 up 1.00000 0
85 1.81934 osd.85 up 1.00000 0
90 3.49219 osd.90 up 1.00000 1.00000
92 3.49219 osd.92 up 1.00000 1.00000
Liebe Grüße,
Christian Theune
--
Christian Theune · ct@xxxxxxxxxxxxxxx · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
Attachment:
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
