Re: Switch to replica 3

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 2017-11-20 3:39 am, Matteo Dacrema wrote:
Yes I mean the existing Cluster.
SSDs are on a fully separate pool.
Cluster is not busy during recovery and deep scrubs but I think it’s
better to limit replication in some way when switching to replica 3.

My question is to understand if I need to set some options parameters
to limit the impact of the creation of new objects.I’m also concerned
about disk filling up during recovery because of inefficient data
balancing.

You can try using osd_recovery_sleep to slow down the backfilling so it does not cause the client io to hang.

ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"



Here osd tree

ID  WEIGHT    TYPE NAME        UP/DOWN REWEIGHT PRIMARY-AFFINITY
-10  19.69994 root ssd
-11   5.06998     host ceph101
166   0.98999         osd.166       up  1.00000          1.00000
167   1.00000         osd.167       up  1.00000          1.00000
168   1.00000         osd.168       up  1.00000          1.00000
169   1.07999         osd.169       up  1.00000          1.00000
170   1.00000         osd.170       up  1.00000          1.00000
-12   4.92998     host ceph102
171   0.98000         osd.171       up  1.00000          1.00000
172   0.92999         osd.172       up  1.00000          1.00000
173   0.98000         osd.173       up  1.00000          1.00000
174   1.00000         osd.174       up  1.00000          1.00000
175   1.03999         osd.175       up  1.00000          1.00000
-13   4.69998     host ceph103
176   0.84999         osd.176       up  1.00000          1.00000
177   0.84999         osd.177       up  1.00000          1.00000
178   1.00000         osd.178       up  1.00000          1.00000
179   1.00000         osd.179       up  1.00000          1.00000
180   1.00000         osd.180       up  1.00000          1.00000
-14   5.00000     host ceph104
181   1.00000         osd.181       up  1.00000          1.00000
182   1.00000         osd.182       up  1.00000          1.00000
183   1.00000         osd.183       up  1.00000          1.00000
184   1.00000         osd.184       up  1.00000          1.00000
185   1.00000         osd.185       up  1.00000          1.00000
 -1 185.19835 root default
 -2  18.39980     host ceph001
 63   0.79999         osd.63        up  1.00000          1.00000
 64   0.79999         osd.64        up  1.00000          1.00000
 65   0.79999         osd.65        up  1.00000          1.00000
146   0.70000         osd.146       up  1.00000          1.00000
147   0.70000         osd.147       up  1.00000          1.00000
148   0.90999         osd.148       up  1.00000          1.00000
149   0.79999         osd.149       up  1.00000          1.00000
150   0.79999         osd.150       up  1.00000          1.00000
151   0.79999         osd.151       up  1.00000          1.00000
152   0.79999         osd.152       up  1.00000          1.00000
153   0.70000         osd.153       up  1.00000          1.00000
154   0.70000         osd.154       up  1.00000          1.00000
155   0.89999         osd.155       up  1.00000          1.00000
156   0.84999         osd.156       up  1.00000          1.00000
157   0.70000         osd.157       up  1.00000          1.00000
158   0.70000         osd.158       up  1.00000          1.00000
159   0.84999         osd.159       up  1.00000          1.00000
160   0.90999         osd.160       up  1.00000          1.00000
161   0.90999         osd.161       up  1.00000          1.00000
162   0.90999         osd.162       up  1.00000          1.00000
163   0.79999         osd.163       up  1.00000          1.00000
164   0.90999         osd.164       up  1.00000          1.00000
165   0.64999         osd.165       up  1.00000          1.00000
 -3  19.41982     host ceph002
 23   0.79999         osd.23        up  1.00000          1.00000
 24   0.70000         osd.24        up  1.00000          1.00000
 25   0.90999         osd.25        up  1.00000          1.00000
 26   0.59999         osd.26        up  1.00000          1.00000
 27   0.95000         osd.27        up  1.00000          1.00000
 28   0.64999         osd.28        up  1.00000          1.00000
 29   0.75000         osd.29        up  1.00000          1.00000
 30   0.89999         osd.30        up  1.00000          1.00000
 31   0.90999         osd.31        up  1.00000          1.00000
 32   0.90999         osd.32        up  1.00000          1.00000
 33   0.89999         osd.33        up  1.00000          1.00000
 34   0.90999         osd.34        up  1.00000          1.00000
 35   0.90999         osd.35        up  1.00000          1.00000
 36   0.84999         osd.36        up  1.00000          1.00000
 37   0.89999         osd.37        up  1.00000          1.00000
 38   1.00000         osd.38        up  1.00000          1.00000
 39   0.79999         osd.39        up  1.00000          1.00000
 40   0.90999         osd.40        up  1.00000          1.00000
 41   0.84999         osd.41        up  1.00000          1.00000
 42   0.84999         osd.42        up  1.00000          1.00000
 43   0.90999         osd.43        up  1.00000          1.00000
 44   0.75000         osd.44        up  1.00000          1.00000
 45   0.79999         osd.45        up  1.00000          1.00000
 -4  16.78984     host ceph003
 20   1.00000         osd.20        up  1.00000          1.00000
 21   0.90999         osd.21        up  1.00000          1.00000
 22   0.90999         osd.22        up  1.00000          1.00000
 46   0.79999         osd.46        up  1.00000          1.00000
 47   0.84999         osd.47        up  1.00000          1.00000
 48   0.75000         osd.48        up  1.00000          1.00000
 49   0.90999         osd.49        up  1.00000          1.00000
 50   0.90999         osd.50        up  1.00000          1.00000
 51   0.90999         osd.51        up  1.00000          1.00000
 52   0.90999         osd.52        up  1.00000          1.00000
 53   0.70000         osd.53        up  1.00000          1.00000
 54   0.70000         osd.54        up  1.00000          1.00000
 55   0.90999         osd.55        up  1.00000          1.00000
 56   0.90999         osd.56        up  1.00000          1.00000
 57   0.75000         osd.57        up  1.00000          1.00000
 58   0.79999         osd.58        up  1.00000          1.00000
 59   0.70000         osd.59        up  1.00000          1.00000
 60   0.90999         osd.60        up  1.00000          1.00000
 61   0.75000         osd.61        up  1.00000          1.00000
 62   0.79999         osd.62        up  1.00000          1.00000
 -5  16.30986     host ceph004
 66   0.95000         osd.66        up  1.00000          1.00000
 67   0.79999         osd.67        up  1.00000          1.00000
 68   0.84999         osd.68        up  1.00000          1.00000
 69   0.84999         osd.69        up  1.00000          1.00000
 70   0.70000         osd.70        up  1.00000          1.00000
 71   0.75000         osd.71        up  1.00000          1.00000
 72   0.64999         osd.72        up  1.00000          1.00000
 73   0.90999         osd.73        up  1.00000          1.00000
 74   0.75000         osd.74        up  1.00000          1.00000
 75   0.81999         osd.75        up  1.00000          1.00000
 76   0.90999         osd.76        up  1.00000          1.00000
 77   0.79999         osd.77        up  1.00000          1.00000
 78   0.59999         osd.78        up  1.00000          1.00000
 79   1.00000         osd.79        up  1.00000          1.00000
 80   0.90999         osd.80        up  1.00000          1.00000
 81   0.79999         osd.81        up  1.00000          1.00000
 82   0.89999         osd.82        up  1.00000          1.00000
 83   0.70000         osd.83        up  1.00000          1.00000
 84   0.75000         osd.84        up  1.00000          1.00000
 85   0.90999         osd.85        up  1.00000          1.00000
 -6  16.71983     host ceph005
 86   0.75000         osd.86        up  1.00000          1.00000
 87   0.90999         osd.87        up  1.00000          1.00000
 88   0.89999         osd.88        up  1.00000          1.00000
 89   0.90999         osd.89        up  1.00000          1.00000
 90   0.90999         osd.90        up  1.00000          1.00000
 91   0.89999         osd.91        up  1.00000          1.00000
 92   0.89999         osd.92        up  1.00000          1.00000
 93   0.90999         osd.93        up  1.00000          1.00000
 94   0.84999         osd.94        up  1.00000          1.00000
 95   0.90999         osd.95        up  1.00000          1.00000
 96   0.50000         osd.96        up  1.00000          1.00000
 97   0.79999         osd.97        up  1.00000          1.00000
 98   0.59999         osd.98        up  1.00000          1.00000
 99   0.75000         osd.99        up  1.00000          1.00000
100   1.00000         osd.100       up  1.00000          1.00000
101   0.79999         osd.101       up  1.00000          1.00000
102   0.90999         osd.102       up  1.00000          1.00000
103   0.79999         osd.103       up  1.00000          1.00000
104   0.90999         osd.104       up  1.00000          1.00000
105   0.79999         osd.105       up  1.00000          1.00000
 -7  17.19980     host ceph006
106   0.90999         osd.106       up  1.00000          1.00000
107   0.89999         osd.107       up  1.00000          1.00000
108   0.70000         osd.108       up  1.00000          1.00000
109   0.84999         osd.109       up  1.00000          1.00000
110   0.79999         osd.110       up  1.00000          1.00000
111   0.79999         osd.111       up  1.00000          1.00000
112   0.79999         osd.112       up  1.00000          1.00000
113   0.90999         osd.113       up  1.00000          1.00000
115   0.90999         osd.115       up  1.00000          1.00000
116   0.90999         osd.116       up  1.00000          1.00000
117   0.90999         osd.117       up  1.00000          1.00000
118   0.90999         osd.118       up  1.00000          1.00000
119   0.59999         osd.119       up  1.00000          1.00000
120   0.90999         osd.120       up  1.00000          1.00000
121   0.90999         osd.121       up  1.00000          1.00000
122   0.90999         osd.122       up  1.00000          1.00000
123   0.79999         osd.123       up  1.00000          1.00000
124   0.75000         osd.124       up  1.00000          1.00000
125   1.09999         osd.125       up  1.00000          1.00000
114   0.90999         osd.114       up  1.00000          1.00000
 -8  16.61984     host ceph007
126   0.90999         osd.126       up  1.00000          1.00000
127   0.75000         osd.127       up  1.00000          1.00000
128   0.84999         osd.128       up  1.00000          1.00000
129   0.90999         osd.129       up  1.00000          1.00000
130   0.70000         osd.130       up  1.00000          1.00000
131   0.78000         osd.131       up  1.00000          1.00000
132   0.79999         osd.132       up  1.00000          1.00000
133   0.54999         osd.133       up  1.00000          1.00000
134   0.79999         osd.134       up  1.00000          1.00000
135   0.87000         osd.135       up  1.00000          1.00000
136   0.90999         osd.136       up  1.00000          1.00000
137   0.90999         osd.137       up  1.00000          1.00000
138   0.90999         osd.138       up  1.00000          1.00000
139   0.89999         osd.139       up  1.00000          1.00000
140   0.75000         osd.140       up  1.00000          1.00000
141   0.75000         osd.141       up  1.00000          1.00000
142   0.90999         osd.142       up  1.00000          1.00000
143   0.84999         osd.143       up  1.00000          1.00000
145   0.90999         osd.145       up  1.00000          1.00000
144   0.89999         osd.144       up  1.00000          1.00000
 -9  16.93982     host ceph008
  0   0.90999         osd.0         up  1.00000          1.00000
  1   0.70000         osd.1         up  1.00000          1.00000
  2   0.90999         osd.2         up  1.00000          1.00000
  3   0.90999         osd.3         up  1.00000          1.00000
  4   0.79999         osd.4         up  1.00000          1.00000
  5   0.64999         osd.5         up  1.00000          1.00000
  6   0.64999         osd.6         up  1.00000          1.00000
  7   0.90999         osd.7         up  1.00000          1.00000
  8   0.79999         osd.8         up  1.00000          1.00000
  9   0.90999         osd.9         up  1.00000          1.00000
 10   0.75000         osd.10        up  1.00000          1.00000
 11   0.90999         osd.11        up  1.00000          1.00000
 12   0.89999         osd.12        up  1.00000          1.00000
 13   0.75000         osd.13        up  1.00000          1.00000
 14   1.09999         osd.14        up  1.00000          1.00000
 15   0.84999         osd.15        up  1.00000          1.00000
 16   0.90999         osd.16        up  1.00000          1.00000
 17   0.90999         osd.17        up  1.00000          1.00000
 18   0.79999         osd.18        up  1.00000          1.00000
 19   0.90999         osd.19        up  1.00000          1.00000
-15  28.79987     host ceph009
186   1.00000         osd.186       up  1.00000          1.00000
187   1.00000         osd.187       up  1.00000          1.00000
188   1.00000         osd.188       up  1.00000          1.00000
189   1.00000         osd.189       up  1.00000          1.00000
190   1.00000         osd.190       up  1.00000          1.00000
191   1.00000         osd.191       up  1.00000          1.00000
192   1.00000         osd.192       up  1.00000          1.00000
193   1.00000         osd.193       up  1.00000          1.00000
194   1.00000         osd.194       up  1.00000          1.00000
195   1.79999         osd.195       up  1.00000          1.00000
196   1.79999         osd.196       up  1.00000          1.00000
197   1.79999         osd.197       up  1.00000          1.00000
198   1.79999         osd.198       up  1.00000          1.00000
199   1.79999         osd.199       up  1.00000          1.00000
200   1.79999         osd.200       up  1.00000          1.00000
201   1.79999         osd.201       up  1.00000          1.00000
202   1.79999         osd.202       up  1.00000          1.00000
203   1.79999         osd.203       up  1.00000          1.00000
204   1.79999         osd.204       up  1.00000          1.00000
205   1.79999         osd.205       up  1.00000          1.00000
-16  17.99988     host ceph010
206   1.79999         osd.206       up  1.00000          1.00000
207   1.79999         osd.207       up  1.00000          1.00000
208   1.79999         osd.208       up  1.00000          1.00000
209   1.79999         osd.209       up  1.00000          1.00000
210   1.79999         osd.210       up  1.00000          1.00000
211   1.79999         osd.211       up  1.00000          1.00000
212   1.79999         osd.212       up  1.00000          1.00000
213   1.79999         osd.213       up  1.00000          1.00000
214   1.79999         osd.214       up  1.00000          1.00000
215   1.79999         osd.215       up  1.00000          1.00000



Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer <chibi@xxxxxxx> ha scritto:


Hello,

On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote:

Hi,

I need to switch a cluster of over 200 OSDs from replica 2 to replica 3
I presume this means the existing cluster and not adding 100 OSDs...

There are two different crush maps for HDD and SSDs also mapped to two different pools.

Is there a best practice to use? Can this provoke troubles?

Are your SSDs a cache-tier or are they a fully separate pool?

As for troubles, how busy is your cluster during the recovery of failed
OSDs or deep scrubs?

There are 2 things to consider here:

1. The re-balancing and additional replication of all the data, which you
can control/ease by the various knobs present. Ceph version matters to
which are relevant/useful. It shouldn't impact things too much, unless
your cluster was at the very edge of it's capacity anyway.

2. The little detail that after 1) is done, your cluster will be
noticeably slower than before, especially in the latency department.
In short, you don't just need to have the disk space to go 3x, but also
enough IOPS/bandwidth reserves.

Christian

Thank you
Matteo
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Christian Balzer        Network/Systems Engineer
chibi@xxxxxxx   	Rakuten Communications

--
Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non infetto.
Seguire il link qui sotto per segnalarlo come spam:
http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC



_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux