-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Did you try to deep-scrub the PG after copying it to 29? - ---------------- Robert LeBlanc GPG Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Jun 4, 2015 at 10:26 PM, Aaron Ten Clay wrote: > Hi Cephers, > > I recently had a power problem and the entire cluster was brought down, came > up, went down, and came up again. Afterword, 3 OSDs were mostly dead (HDD > failures). Luckily (I think) the drives were alive enough that I could copy > the data off and leave the journal alone. > > Since my pool "data" size is 3... of course a couple of placement groups > were only on those three drives. > > Now I've added 4 new OSDs, and everything has recovered, except pg 0.f3. > When I query the pg, I see the cluster is looking for OSD 14 or 23 because > one of them maybe_went_rw. (5, 14, and 23 are now kaput and "ceph osd lost > --yes-i-really-mean-it") > > Ceph indicates OSD 29 is now the primary for pg 0.f3. I copied all the data > to the appropriate directory, started OSD.29 again, and here is where my > question comes in: > > How do I convince the cluster that it's okay to bring 0.f3 'up' and backfill > to the other OSDs from 29? (I could even manually backfill 15 and 22, but I > suspect the cluster will still think there's a problem) > > 'ceph health detail' shows this about 0.f3: > pg 0.f3 is incomplete, acting [29,22,15] (reducing pool data min_size from 2 > may help; search ceph.com/docs for 'incomplete') > > Thanks in advance! > -Aaron > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -----BEGIN PGP SIGNATURE----- Version: Mailvelope v0.13.1 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJVcee3CRDmVDuy+mK58QAA0g4QAIGmDnQcR2LtMr9YqJfz mUwC1ZN35WCzFn8varbTcJZqW0MEHQZ3vP6EqWcoPqQhc6ehHi8+onjTTmDf 1CLMAyqxh/rRglqYP9EnWXdzMmOaQCDC5ImOLD79tvjxDAsRZ1TPwtPWsj3S CxaitKF13dot31AftUjTUsZ8eTjNyCT1VIQJVEiHxVtgQfOlEGNaQnYDZ955 eJKsKTBoTlU1nZe0jKCefac8ZgJ32ngGu4KGyfUEx4MQsRWx0Uv9+71OiTJ4 E3mxxtR1OBhcbAK6syqm1sh8ELa8fMOBTZzC9nauYpYfhvnW5yk5PmDrUdri gGoQozqspIPmaa9fXjO5Q8+L86mhG69CnlRU7zD2frzLp6RCT6dWQhhTuHt5 oPWbdoY6dWY93nJPjGo3Dh4WnFR1SAezn1STOI1rtIQGu+uvD5Joc/6QUuUS R9mSC3XycUX1YfGGetsaz1PZhBV0CEytrnOh22CM907luvSubXyd5Hrddu3p zNQ1xUz5IFsaDPWgk27k3vY98CfZy+EJJ7fJvlC6gU6NHX0Y7YrU6kzGauiQ 0NKMfjjSsaxaPnHKtPbQ8OkX4j1hvXsgHKsClLNaVQHvQFlSB5rScQZ7rmYV I/UWxhIz70q467J0eA0+kCqTsTEw6VJ62bdoWrd2MkAwd4WpRCA/VtK73B6+ 9ByE =ODgo -----END PGP SIGNATURE----- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |