On Mon, 20 Nov 2017 10:35:36 -0800 Chris Taylor wrote: > On 2017-11-20 3:39 am, Matteo Dacrema wrote: > > Yes I mean the existing Cluster. > > SSDs are on a fully separate pool. > > Cluster is not busy during recovery and deep scrubs but I think it’s > > better to limit replication in some way when switching to replica 3. > > > > My question is to understand if I need to set some options parameters > > to limit the impact of the creation of new objects.I’m also concerned > > about disk filling up during recovery because of inefficient data > > balancing. > > You can try using osd_recovery_sleep to slow down the backfilling so it > does not cause the client io to hang. > > ceph tell osd.* injectargs "--osd_recovery_sleep 0.1" > Which is one of the things that is version specific and we don't know the version yet. The above will work with Hammer and should again with Luminous, but not so much with the unified queue bits inbetween. Christian > > > > > Here osd tree > > > > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > > -10 19.69994 root ssd > > -11 5.06998 host ceph101 > > 166 0.98999 osd.166 up 1.00000 1.00000 > > 167 1.00000 osd.167 up 1.00000 1.00000 > > 168 1.00000 osd.168 up 1.00000 1.00000 > > 169 1.07999 osd.169 up 1.00000 1.00000 > > 170 1.00000 osd.170 up 1.00000 1.00000 > > -12 4.92998 host ceph102 > > 171 0.98000 osd.171 up 1.00000 1.00000 > > 172 0.92999 osd.172 up 1.00000 1.00000 > > 173 0.98000 osd.173 up 1.00000 1.00000 > > 174 1.00000 osd.174 up 1.00000 1.00000 > > 175 1.03999 osd.175 up 1.00000 1.00000 > > -13 4.69998 host ceph103 > > 176 0.84999 osd.176 up 1.00000 1.00000 > > 177 0.84999 osd.177 up 1.00000 1.00000 > > 178 1.00000 osd.178 up 1.00000 1.00000 > > 179 1.00000 osd.179 up 1.00000 1.00000 > > 180 1.00000 osd.180 up 1.00000 1.00000 > > -14 5.00000 host ceph104 > > 181 1.00000 osd.181 up 1.00000 1.00000 > > 182 1.00000 osd.182 up 1.00000 1.00000 > > 183 1.00000 osd.183 up 1.00000 1.00000 > > 184 1.00000 osd.184 up 1.00000 1.00000 > > 185 1.00000 osd.185 up 1.00000 1.00000 > > -1 185.19835 root default > > -2 18.39980 host ceph001 > > 63 0.79999 osd.63 up 1.00000 1.00000 > > 64 0.79999 osd.64 up 1.00000 1.00000 > > 65 0.79999 osd.65 up 1.00000 1.00000 > > 146 0.70000 osd.146 up 1.00000 1.00000 > > 147 0.70000 osd.147 up 1.00000 1.00000 > > 148 0.90999 osd.148 up 1.00000 1.00000 > > 149 0.79999 osd.149 up 1.00000 1.00000 > > 150 0.79999 osd.150 up 1.00000 1.00000 > > 151 0.79999 osd.151 up 1.00000 1.00000 > > 152 0.79999 osd.152 up 1.00000 1.00000 > > 153 0.70000 osd.153 up 1.00000 1.00000 > > 154 0.70000 osd.154 up 1.00000 1.00000 > > 155 0.89999 osd.155 up 1.00000 1.00000 > > 156 0.84999 osd.156 up 1.00000 1.00000 > > 157 0.70000 osd.157 up 1.00000 1.00000 > > 158 0.70000 osd.158 up 1.00000 1.00000 > > 159 0.84999 osd.159 up 1.00000 1.00000 > > 160 0.90999 osd.160 up 1.00000 1.00000 > > 161 0.90999 osd.161 up 1.00000 1.00000 > > 162 0.90999 osd.162 up 1.00000 1.00000 > > 163 0.79999 osd.163 up 1.00000 1.00000 > > 164 0.90999 osd.164 up 1.00000 1.00000 > > 165 0.64999 osd.165 up 1.00000 1.00000 > > -3 19.41982 host ceph002 > > 23 0.79999 osd.23 up 1.00000 1.00000 > > 24 0.70000 osd.24 up 1.00000 1.00000 > > 25 0.90999 osd.25 up 1.00000 1.00000 > > 26 0.59999 osd.26 up 1.00000 1.00000 > > 27 0.95000 osd.27 up 1.00000 1.00000 > > 28 0.64999 osd.28 up 1.00000 1.00000 > > 29 0.75000 osd.29 up 1.00000 1.00000 > > 30 0.89999 osd.30 up 1.00000 1.00000 > > 31 0.90999 osd.31 up 1.00000 1.00000 > > 32 0.90999 osd.32 up 1.00000 1.00000 > > 33 0.89999 osd.33 up 1.00000 1.00000 > > 34 0.90999 osd.34 up 1.00000 1.00000 > > 35 0.90999 osd.35 up 1.00000 1.00000 > > 36 0.84999 osd.36 up 1.00000 1.00000 > > 37 0.89999 osd.37 up 1.00000 1.00000 > > 38 1.00000 osd.38 up 1.00000 1.00000 > > 39 0.79999 osd.39 up 1.00000 1.00000 > > 40 0.90999 osd.40 up 1.00000 1.00000 > > 41 0.84999 osd.41 up 1.00000 1.00000 > > 42 0.84999 osd.42 up 1.00000 1.00000 > > 43 0.90999 osd.43 up 1.00000 1.00000 > > 44 0.75000 osd.44 up 1.00000 1.00000 > > 45 0.79999 osd.45 up 1.00000 1.00000 > > -4 16.78984 host ceph003 > > 20 1.00000 osd.20 up 1.00000 1.00000 > > 21 0.90999 osd.21 up 1.00000 1.00000 > > 22 0.90999 osd.22 up 1.00000 1.00000 > > 46 0.79999 osd.46 up 1.00000 1.00000 > > 47 0.84999 osd.47 up 1.00000 1.00000 > > 48 0.75000 osd.48 up 1.00000 1.00000 > > 49 0.90999 osd.49 up 1.00000 1.00000 > > 50 0.90999 osd.50 up 1.00000 1.00000 > > 51 0.90999 osd.51 up 1.00000 1.00000 > > 52 0.90999 osd.52 up 1.00000 1.00000 > > 53 0.70000 osd.53 up 1.00000 1.00000 > > 54 0.70000 osd.54 up 1.00000 1.00000 > > 55 0.90999 osd.55 up 1.00000 1.00000 > > 56 0.90999 osd.56 up 1.00000 1.00000 > > 57 0.75000 osd.57 up 1.00000 1.00000 > > 58 0.79999 osd.58 up 1.00000 1.00000 > > 59 0.70000 osd.59 up 1.00000 1.00000 > > 60 0.90999 osd.60 up 1.00000 1.00000 > > 61 0.75000 osd.61 up 1.00000 1.00000 > > 62 0.79999 osd.62 up 1.00000 1.00000 > > -5 16.30986 host ceph004 > > 66 0.95000 osd.66 up 1.00000 1.00000 > > 67 0.79999 osd.67 up 1.00000 1.00000 > > 68 0.84999 osd.68 up 1.00000 1.00000 > > 69 0.84999 osd.69 up 1.00000 1.00000 > > 70 0.70000 osd.70 up 1.00000 1.00000 > > 71 0.75000 osd.71 up 1.00000 1.00000 > > 72 0.64999 osd.72 up 1.00000 1.00000 > > 73 0.90999 osd.73 up 1.00000 1.00000 > > 74 0.75000 osd.74 up 1.00000 1.00000 > > 75 0.81999 osd.75 up 1.00000 1.00000 > > 76 0.90999 osd.76 up 1.00000 1.00000 > > 77 0.79999 osd.77 up 1.00000 1.00000 > > 78 0.59999 osd.78 up 1.00000 1.00000 > > 79 1.00000 osd.79 up 1.00000 1.00000 > > 80 0.90999 osd.80 up 1.00000 1.00000 > > 81 0.79999 osd.81 up 1.00000 1.00000 > > 82 0.89999 osd.82 up 1.00000 1.00000 > > 83 0.70000 osd.83 up 1.00000 1.00000 > > 84 0.75000 osd.84 up 1.00000 1.00000 > > 85 0.90999 osd.85 up 1.00000 1.00000 > > -6 16.71983 host ceph005 > > 86 0.75000 osd.86 up 1.00000 1.00000 > > 87 0.90999 osd.87 up 1.00000 1.00000 > > 88 0.89999 osd.88 up 1.00000 1.00000 > > 89 0.90999 osd.89 up 1.00000 1.00000 > > 90 0.90999 osd.90 up 1.00000 1.00000 > > 91 0.89999 osd.91 up 1.00000 1.00000 > > 92 0.89999 osd.92 up 1.00000 1.00000 > > 93 0.90999 osd.93 up 1.00000 1.00000 > > 94 0.84999 osd.94 up 1.00000 1.00000 > > 95 0.90999 osd.95 up 1.00000 1.00000 > > 96 0.50000 osd.96 up 1.00000 1.00000 > > 97 0.79999 osd.97 up 1.00000 1.00000 > > 98 0.59999 osd.98 up 1.00000 1.00000 > > 99 0.75000 osd.99 up 1.00000 1.00000 > > 100 1.00000 osd.100 up 1.00000 1.00000 > > 101 0.79999 osd.101 up 1.00000 1.00000 > > 102 0.90999 osd.102 up 1.00000 1.00000 > > 103 0.79999 osd.103 up 1.00000 1.00000 > > 104 0.90999 osd.104 up 1.00000 1.00000 > > 105 0.79999 osd.105 up 1.00000 1.00000 > > -7 17.19980 host ceph006 > > 106 0.90999 osd.106 up 1.00000 1.00000 > > 107 0.89999 osd.107 up 1.00000 1.00000 > > 108 0.70000 osd.108 up 1.00000 1.00000 > > 109 0.84999 osd.109 up 1.00000 1.00000 > > 110 0.79999 osd.110 up 1.00000 1.00000 > > 111 0.79999 osd.111 up 1.00000 1.00000 > > 112 0.79999 osd.112 up 1.00000 1.00000 > > 113 0.90999 osd.113 up 1.00000 1.00000 > > 115 0.90999 osd.115 up 1.00000 1.00000 > > 116 0.90999 osd.116 up 1.00000 1.00000 > > 117 0.90999 osd.117 up 1.00000 1.00000 > > 118 0.90999 osd.118 up 1.00000 1.00000 > > 119 0.59999 osd.119 up 1.00000 1.00000 > > 120 0.90999 osd.120 up 1.00000 1.00000 > > 121 0.90999 osd.121 up 1.00000 1.00000 > > 122 0.90999 osd.122 up 1.00000 1.00000 > > 123 0.79999 osd.123 up 1.00000 1.00000 > > 124 0.75000 osd.124 up 1.00000 1.00000 > > 125 1.09999 osd.125 up 1.00000 1.00000 > > 114 0.90999 osd.114 up 1.00000 1.00000 > > -8 16.61984 host ceph007 > > 126 0.90999 osd.126 up 1.00000 1.00000 > > 127 0.75000 osd.127 up 1.00000 1.00000 > > 128 0.84999 osd.128 up 1.00000 1.00000 > > 129 0.90999 osd.129 up 1.00000 1.00000 > > 130 0.70000 osd.130 up 1.00000 1.00000 > > 131 0.78000 osd.131 up 1.00000 1.00000 > > 132 0.79999 osd.132 up 1.00000 1.00000 > > 133 0.54999 osd.133 up 1.00000 1.00000 > > 134 0.79999 osd.134 up 1.00000 1.00000 > > 135 0.87000 osd.135 up 1.00000 1.00000 > > 136 0.90999 osd.136 up 1.00000 1.00000 > > 137 0.90999 osd.137 up 1.00000 1.00000 > > 138 0.90999 osd.138 up 1.00000 1.00000 > > 139 0.89999 osd.139 up 1.00000 1.00000 > > 140 0.75000 osd.140 up 1.00000 1.00000 > > 141 0.75000 osd.141 up 1.00000 1.00000 > > 142 0.90999 osd.142 up 1.00000 1.00000 > > 143 0.84999 osd.143 up 1.00000 1.00000 > > 145 0.90999 osd.145 up 1.00000 1.00000 > > 144 0.89999 osd.144 up 1.00000 1.00000 > > -9 16.93982 host ceph008 > > 0 0.90999 osd.0 up 1.00000 1.00000 > > 1 0.70000 osd.1 up 1.00000 1.00000 > > 2 0.90999 osd.2 up 1.00000 1.00000 > > 3 0.90999 osd.3 up 1.00000 1.00000 > > 4 0.79999 osd.4 up 1.00000 1.00000 > > 5 0.64999 osd.5 up 1.00000 1.00000 > > 6 0.64999 osd.6 up 1.00000 1.00000 > > 7 0.90999 osd.7 up 1.00000 1.00000 > > 8 0.79999 osd.8 up 1.00000 1.00000 > > 9 0.90999 osd.9 up 1.00000 1.00000 > > 10 0.75000 osd.10 up 1.00000 1.00000 > > 11 0.90999 osd.11 up 1.00000 1.00000 > > 12 0.89999 osd.12 up 1.00000 1.00000 > > 13 0.75000 osd.13 up 1.00000 1.00000 > > 14 1.09999 osd.14 up 1.00000 1.00000 > > 15 0.84999 osd.15 up 1.00000 1.00000 > > 16 0.90999 osd.16 up 1.00000 1.00000 > > 17 0.90999 osd.17 up 1.00000 1.00000 > > 18 0.79999 osd.18 up 1.00000 1.00000 > > 19 0.90999 osd.19 up 1.00000 1.00000 > > -15 28.79987 host ceph009 > > 186 1.00000 osd.186 up 1.00000 1.00000 > > 187 1.00000 osd.187 up 1.00000 1.00000 > > 188 1.00000 osd.188 up 1.00000 1.00000 > > 189 1.00000 osd.189 up 1.00000 1.00000 > > 190 1.00000 osd.190 up 1.00000 1.00000 > > 191 1.00000 osd.191 up 1.00000 1.00000 > > 192 1.00000 osd.192 up 1.00000 1.00000 > > 193 1.00000 osd.193 up 1.00000 1.00000 > > 194 1.00000 osd.194 up 1.00000 1.00000 > > 195 1.79999 osd.195 up 1.00000 1.00000 > > 196 1.79999 osd.196 up 1.00000 1.00000 > > 197 1.79999 osd.197 up 1.00000 1.00000 > > 198 1.79999 osd.198 up 1.00000 1.00000 > > 199 1.79999 osd.199 up 1.00000 1.00000 > > 200 1.79999 osd.200 up 1.00000 1.00000 > > 201 1.79999 osd.201 up 1.00000 1.00000 > > 202 1.79999 osd.202 up 1.00000 1.00000 > > 203 1.79999 osd.203 up 1.00000 1.00000 > > 204 1.79999 osd.204 up 1.00000 1.00000 > > 205 1.79999 osd.205 up 1.00000 1.00000 > > -16 17.99988 host ceph010 > > 206 1.79999 osd.206 up 1.00000 1.00000 > > 207 1.79999 osd.207 up 1.00000 1.00000 > > 208 1.79999 osd.208 up 1.00000 1.00000 > > 209 1.79999 osd.209 up 1.00000 1.00000 > > 210 1.79999 osd.210 up 1.00000 1.00000 > > 211 1.79999 osd.211 up 1.00000 1.00000 > > 212 1.79999 osd.212 up 1.00000 1.00000 > > 213 1.79999 osd.213 up 1.00000 1.00000 > > 214 1.79999 osd.214 up 1.00000 1.00000 > > 215 1.79999 osd.215 up 1.00000 1.00000 > > > > > > > >> Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer > >> <chibi@xxxxxxx> ha scritto: > >> > >> > >> Hello, > >> > >> On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote: > >> > >>> Hi, > >>> > >>> I need to switch a cluster of over 200 OSDs from replica 2 to replica > >>> 3 > >> I presume this means the existing cluster and not adding 100 OSDs... > >> > >>> There are two different crush maps for HDD and SSDs also mapped to > >>> two different pools. > >>> > >>> Is there a best practice to use? Can this provoke troubles? > >>> > >> Are your SSDs a cache-tier or are they a fully separate pool? > >> > >> As for troubles, how busy is your cluster during the recovery of > >> failed > >> OSDs or deep scrubs? > >> > >> There are 2 things to consider here: > >> > >> 1. The re-balancing and additional replication of all the data, which > >> you > >> can control/ease by the various knobs present. Ceph version matters to > >> which are relevant/useful. It shouldn't impact things too much, unless > >> your cluster was at the very edge of it's capacity anyway. > >> > >> 2. The little detail that after 1) is done, your cluster will be > >> noticeably slower than before, especially in the latency department. > >> In short, you don't just need to have the disk space to go 3x, but > >> also > >> enough IOPS/bandwidth reserves. > >> > >> Christian > >> > >>> Thank you > >>> Matteo > >>> _______________________________________________ > >>> ceph-users mailing list > >>> ceph-users@xxxxxxxxxxxxxx > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >> > >> > >> -- > >> Christian Balzer Network/Systems Engineer > >> chibi@xxxxxxx Rakuten Communications > >> > >> -- > >> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato > >> non infetto. > >> Seguire il link qui sotto per segnalarlo come spam: > >> http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC > >> > >> > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Rakuten Communications _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |