On 2017-11-20 3:39 am, Matteo Dacrema wrote:
Yes I mean the existing Cluster.
SSDs are on a fully separate pool.
Cluster is not busy during recovery and deep scrubs but I think it’s
better to limit replication in some way when switching to replica 3.
My question is to understand if I need to set some options parameters
to limit the impact of the creation of new objects.I’m also concerned
about disk filling up during recovery because of inefficient data
balancing.
You can try using osd_recovery_sleep to slow down the backfilling so it
does not cause the client io to hang.
ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"
Here osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-10 19.69994 root ssd
-11 5.06998 host ceph101
166 0.98999 osd.166 up 1.00000 1.00000
167 1.00000 osd.167 up 1.00000 1.00000
168 1.00000 osd.168 up 1.00000 1.00000
169 1.07999 osd.169 up 1.00000 1.00000
170 1.00000 osd.170 up 1.00000 1.00000
-12 4.92998 host ceph102
171 0.98000 osd.171 up 1.00000 1.00000
172 0.92999 osd.172 up 1.00000 1.00000
173 0.98000 osd.173 up 1.00000 1.00000
174 1.00000 osd.174 up 1.00000 1.00000
175 1.03999 osd.175 up 1.00000 1.00000
-13 4.69998 host ceph103
176 0.84999 osd.176 up 1.00000 1.00000
177 0.84999 osd.177 up 1.00000 1.00000
178 1.00000 osd.178 up 1.00000 1.00000
179 1.00000 osd.179 up 1.00000 1.00000
180 1.00000 osd.180 up 1.00000 1.00000
-14 5.00000 host ceph104
181 1.00000 osd.181 up 1.00000 1.00000
182 1.00000 osd.182 up 1.00000 1.00000
183 1.00000 osd.183 up 1.00000 1.00000
184 1.00000 osd.184 up 1.00000 1.00000
185 1.00000 osd.185 up 1.00000 1.00000
-1 185.19835 root default
-2 18.39980 host ceph001
63 0.79999 osd.63 up 1.00000 1.00000
64 0.79999 osd.64 up 1.00000 1.00000
65 0.79999 osd.65 up 1.00000 1.00000
146 0.70000 osd.146 up 1.00000 1.00000
147 0.70000 osd.147 up 1.00000 1.00000
148 0.90999 osd.148 up 1.00000 1.00000
149 0.79999 osd.149 up 1.00000 1.00000
150 0.79999 osd.150 up 1.00000 1.00000
151 0.79999 osd.151 up 1.00000 1.00000
152 0.79999 osd.152 up 1.00000 1.00000
153 0.70000 osd.153 up 1.00000 1.00000
154 0.70000 osd.154 up 1.00000 1.00000
155 0.89999 osd.155 up 1.00000 1.00000
156 0.84999 osd.156 up 1.00000 1.00000
157 0.70000 osd.157 up 1.00000 1.00000
158 0.70000 osd.158 up 1.00000 1.00000
159 0.84999 osd.159 up 1.00000 1.00000
160 0.90999 osd.160 up 1.00000 1.00000
161 0.90999 osd.161 up 1.00000 1.00000
162 0.90999 osd.162 up 1.00000 1.00000
163 0.79999 osd.163 up 1.00000 1.00000
164 0.90999 osd.164 up 1.00000 1.00000
165 0.64999 osd.165 up 1.00000 1.00000
-3 19.41982 host ceph002
23 0.79999 osd.23 up 1.00000 1.00000
24 0.70000 osd.24 up 1.00000 1.00000
25 0.90999 osd.25 up 1.00000 1.00000
26 0.59999 osd.26 up 1.00000 1.00000
27 0.95000 osd.27 up 1.00000 1.00000
28 0.64999 osd.28 up 1.00000 1.00000
29 0.75000 osd.29 up 1.00000 1.00000
30 0.89999 osd.30 up 1.00000 1.00000
31 0.90999 osd.31 up 1.00000 1.00000
32 0.90999 osd.32 up 1.00000 1.00000
33 0.89999 osd.33 up 1.00000 1.00000
34 0.90999 osd.34 up 1.00000 1.00000
35 0.90999 osd.35 up 1.00000 1.00000
36 0.84999 osd.36 up 1.00000 1.00000
37 0.89999 osd.37 up 1.00000 1.00000
38 1.00000 osd.38 up 1.00000 1.00000
39 0.79999 osd.39 up 1.00000 1.00000
40 0.90999 osd.40 up 1.00000 1.00000
41 0.84999 osd.41 up 1.00000 1.00000
42 0.84999 osd.42 up 1.00000 1.00000
43 0.90999 osd.43 up 1.00000 1.00000
44 0.75000 osd.44 up 1.00000 1.00000
45 0.79999 osd.45 up 1.00000 1.00000
-4 16.78984 host ceph003
20 1.00000 osd.20 up 1.00000 1.00000
21 0.90999 osd.21 up 1.00000 1.00000
22 0.90999 osd.22 up 1.00000 1.00000
46 0.79999 osd.46 up 1.00000 1.00000
47 0.84999 osd.47 up 1.00000 1.00000
48 0.75000 osd.48 up 1.00000 1.00000
49 0.90999 osd.49 up 1.00000 1.00000
50 0.90999 osd.50 up 1.00000 1.00000
51 0.90999 osd.51 up 1.00000 1.00000
52 0.90999 osd.52 up 1.00000 1.00000
53 0.70000 osd.53 up 1.00000 1.00000
54 0.70000 osd.54 up 1.00000 1.00000
55 0.90999 osd.55 up 1.00000 1.00000
56 0.90999 osd.56 up 1.00000 1.00000
57 0.75000 osd.57 up 1.00000 1.00000
58 0.79999 osd.58 up 1.00000 1.00000
59 0.70000 osd.59 up 1.00000 1.00000
60 0.90999 osd.60 up 1.00000 1.00000
61 0.75000 osd.61 up 1.00000 1.00000
62 0.79999 osd.62 up 1.00000 1.00000
-5 16.30986 host ceph004
66 0.95000 osd.66 up 1.00000 1.00000
67 0.79999 osd.67 up 1.00000 1.00000
68 0.84999 osd.68 up 1.00000 1.00000
69 0.84999 osd.69 up 1.00000 1.00000
70 0.70000 osd.70 up 1.00000 1.00000
71 0.75000 osd.71 up 1.00000 1.00000
72 0.64999 osd.72 up 1.00000 1.00000
73 0.90999 osd.73 up 1.00000 1.00000
74 0.75000 osd.74 up 1.00000 1.00000
75 0.81999 osd.75 up 1.00000 1.00000
76 0.90999 osd.76 up 1.00000 1.00000
77 0.79999 osd.77 up 1.00000 1.00000
78 0.59999 osd.78 up 1.00000 1.00000
79 1.00000 osd.79 up 1.00000 1.00000
80 0.90999 osd.80 up 1.00000 1.00000
81 0.79999 osd.81 up 1.00000 1.00000
82 0.89999 osd.82 up 1.00000 1.00000
83 0.70000 osd.83 up 1.00000 1.00000
84 0.75000 osd.84 up 1.00000 1.00000
85 0.90999 osd.85 up 1.00000 1.00000
-6 16.71983 host ceph005
86 0.75000 osd.86 up 1.00000 1.00000
87 0.90999 osd.87 up 1.00000 1.00000
88 0.89999 osd.88 up 1.00000 1.00000
89 0.90999 osd.89 up 1.00000 1.00000
90 0.90999 osd.90 up 1.00000 1.00000
91 0.89999 osd.91 up 1.00000 1.00000
92 0.89999 osd.92 up 1.00000 1.00000
93 0.90999 osd.93 up 1.00000 1.00000
94 0.84999 osd.94 up 1.00000 1.00000
95 0.90999 osd.95 up 1.00000 1.00000
96 0.50000 osd.96 up 1.00000 1.00000
97 0.79999 osd.97 up 1.00000 1.00000
98 0.59999 osd.98 up 1.00000 1.00000
99 0.75000 osd.99 up 1.00000 1.00000
100 1.00000 osd.100 up 1.00000 1.00000
101 0.79999 osd.101 up 1.00000 1.00000
102 0.90999 osd.102 up 1.00000 1.00000
103 0.79999 osd.103 up 1.00000 1.00000
104 0.90999 osd.104 up 1.00000 1.00000
105 0.79999 osd.105 up 1.00000 1.00000
-7 17.19980 host ceph006
106 0.90999 osd.106 up 1.00000 1.00000
107 0.89999 osd.107 up 1.00000 1.00000
108 0.70000 osd.108 up 1.00000 1.00000
109 0.84999 osd.109 up 1.00000 1.00000
110 0.79999 osd.110 up 1.00000 1.00000
111 0.79999 osd.111 up 1.00000 1.00000
112 0.79999 osd.112 up 1.00000 1.00000
113 0.90999 osd.113 up 1.00000 1.00000
115 0.90999 osd.115 up 1.00000 1.00000
116 0.90999 osd.116 up 1.00000 1.00000
117 0.90999 osd.117 up 1.00000 1.00000
118 0.90999 osd.118 up 1.00000 1.00000
119 0.59999 osd.119 up 1.00000 1.00000
120 0.90999 osd.120 up 1.00000 1.00000
121 0.90999 osd.121 up 1.00000 1.00000
122 0.90999 osd.122 up 1.00000 1.00000
123 0.79999 osd.123 up 1.00000 1.00000
124 0.75000 osd.124 up 1.00000 1.00000
125 1.09999 osd.125 up 1.00000 1.00000
114 0.90999 osd.114 up 1.00000 1.00000
-8 16.61984 host ceph007
126 0.90999 osd.126 up 1.00000 1.00000
127 0.75000 osd.127 up 1.00000 1.00000
128 0.84999 osd.128 up 1.00000 1.00000
129 0.90999 osd.129 up 1.00000 1.00000
130 0.70000 osd.130 up 1.00000 1.00000
131 0.78000 osd.131 up 1.00000 1.00000
132 0.79999 osd.132 up 1.00000 1.00000
133 0.54999 osd.133 up 1.00000 1.00000
134 0.79999 osd.134 up 1.00000 1.00000
135 0.87000 osd.135 up 1.00000 1.00000
136 0.90999 osd.136 up 1.00000 1.00000
137 0.90999 osd.137 up 1.00000 1.00000
138 0.90999 osd.138 up 1.00000 1.00000
139 0.89999 osd.139 up 1.00000 1.00000
140 0.75000 osd.140 up 1.00000 1.00000
141 0.75000 osd.141 up 1.00000 1.00000
142 0.90999 osd.142 up 1.00000 1.00000
143 0.84999 osd.143 up 1.00000 1.00000
145 0.90999 osd.145 up 1.00000 1.00000
144 0.89999 osd.144 up 1.00000 1.00000
-9 16.93982 host ceph008
0 0.90999 osd.0 up 1.00000 1.00000
1 0.70000 osd.1 up 1.00000 1.00000
2 0.90999 osd.2 up 1.00000 1.00000
3 0.90999 osd.3 up 1.00000 1.00000
4 0.79999 osd.4 up 1.00000 1.00000
5 0.64999 osd.5 up 1.00000 1.00000
6 0.64999 osd.6 up 1.00000 1.00000
7 0.90999 osd.7 up 1.00000 1.00000
8 0.79999 osd.8 up 1.00000 1.00000
9 0.90999 osd.9 up 1.00000 1.00000
10 0.75000 osd.10 up 1.00000 1.00000
11 0.90999 osd.11 up 1.00000 1.00000
12 0.89999 osd.12 up 1.00000 1.00000
13 0.75000 osd.13 up 1.00000 1.00000
14 1.09999 osd.14 up 1.00000 1.00000
15 0.84999 osd.15 up 1.00000 1.00000
16 0.90999 osd.16 up 1.00000 1.00000
17 0.90999 osd.17 up 1.00000 1.00000
18 0.79999 osd.18 up 1.00000 1.00000
19 0.90999 osd.19 up 1.00000 1.00000
-15 28.79987 host ceph009
186 1.00000 osd.186 up 1.00000 1.00000
187 1.00000 osd.187 up 1.00000 1.00000
188 1.00000 osd.188 up 1.00000 1.00000
189 1.00000 osd.189 up 1.00000 1.00000
190 1.00000 osd.190 up 1.00000 1.00000
191 1.00000 osd.191 up 1.00000 1.00000
192 1.00000 osd.192 up 1.00000 1.00000
193 1.00000 osd.193 up 1.00000 1.00000
194 1.00000 osd.194 up 1.00000 1.00000
195 1.79999 osd.195 up 1.00000 1.00000
196 1.79999 osd.196 up 1.00000 1.00000
197 1.79999 osd.197 up 1.00000 1.00000
198 1.79999 osd.198 up 1.00000 1.00000
199 1.79999 osd.199 up 1.00000 1.00000
200 1.79999 osd.200 up 1.00000 1.00000
201 1.79999 osd.201 up 1.00000 1.00000
202 1.79999 osd.202 up 1.00000 1.00000
203 1.79999 osd.203 up 1.00000 1.00000
204 1.79999 osd.204 up 1.00000 1.00000
205 1.79999 osd.205 up 1.00000 1.00000
-16 17.99988 host ceph010
206 1.79999 osd.206 up 1.00000 1.00000
207 1.79999 osd.207 up 1.00000 1.00000
208 1.79999 osd.208 up 1.00000 1.00000
209 1.79999 osd.209 up 1.00000 1.00000
210 1.79999 osd.210 up 1.00000 1.00000
211 1.79999 osd.211 up 1.00000 1.00000
212 1.79999 osd.212 up 1.00000 1.00000
213 1.79999 osd.213 up 1.00000 1.00000
214 1.79999 osd.214 up 1.00000 1.00000
215 1.79999 osd.215 up 1.00000 1.00000
Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer
<chibi@xxxxxxx> ha scritto:
Hello,
On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote:
Hi,
I need to switch a cluster of over 200 OSDs from replica 2 to replica
3
I presume this means the existing cluster and not adding 100 OSDs...
There are two different crush maps for HDD and SSDs also mapped to
two different pools.
Is there a best practice to use? Can this provoke troubles?
Are your SSDs a cache-tier or are they a fully separate pool?
As for troubles, how busy is your cluster during the recovery of
failed
OSDs or deep scrubs?
There are 2 things to consider here:
1. The re-balancing and additional replication of all the data, which
you
can control/ease by the various knobs present. Ceph version matters to
which are relevant/useful. It shouldn't impact things too much, unless
your cluster was at the very edge of it's capacity anyway.
2. The little detail that after 1) is done, your cluster will be
noticeably slower than before, especially in the latency department.
In short, you don't just need to have the disk space to go 3x, but
also
enough IOPS/bandwidth reserves.
Christian
Thank you
Matteo
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Christian Balzer Network/Systems Engineer
chibi@xxxxxxx Rakuten Communications
--
Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato
non infetto.
Seguire il link qui sotto per segnalarlo come spam:
http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
