-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 That seems to have resolved the problem. Thanks! I guess the repair does not increase the epoc of the PG or something so it loses it's brain about it being clean. I'm glad it was a simple resolution. - ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Oct 26, 2015 at 2:32 AM, Dan van der Ster wrote: > On Mon, Oct 26, 2015 at 4:38 AM, Robert LeBlanc wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> I set debug_osd = 20/20 and restarted the primary osd. The logs are at >> http://162.144.87.113/files/ceph-osd.110.log.xz . >> >> The PG in question is 9.e3 and it is one of 15 that have this same >> behavior. The cluster is currently idle. > > We had a similar problem in the past -- see the thread "PGs going > inconsistent after stopping the primary". > The fix is to write to the PG. > > Cheers, Dan > > >> - ---------------- >> Robert LeBlanc >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 >> >> >> On Sun, Oct 25, 2015 at 5:51 PM, Robert LeBlanc wrote: >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA256 >>> >>> I have a 0.94.4 cluster that when I repair/deep-scrub a PG, it comes >>> back clean, but as soon as I restart any OSD that hosts it, it goes >>> back to inconsistent. If I deep-scrub that PG it clears up. >>> >>> I determined that the bad copy was not on the primary and issued a pg >>> repair command. I have shut down and deleted the PG folder on each OSD >>> in turn and let it back fill. I tried taking the primary OSD down and >>> issuing a repair command then. I took an m5sum of all files in the PG >>> directory and compared all files across the OSDs and it came back >>> clean. I shut down each OSD in turn and removed any PG_TEMP >>> directories. I'm just not sure why the cluster is so confused as to >>> the status of this PG. >>> >>> Any ideas? >>> >>> - ---------------- >>> Robert LeBlanc >>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 >>> -----BEGIN PGP SIGNATURE----- >>> Version: Mailvelope v1.2.2 >>> Comment: https://www.mailvelope.com >>> >>> wsFcBAEBCAAQBQJWLWr3CRDmVDuy+mK58QAA4HEP/361NhUXujdrKr9xa4d/ >>> Lr/zMCbppT7uof3BueLkkIF2erun19ENNLZV5ehyzcbeWlHVA3UEzJaZlaew >>> eQrmFN0TKk2BFtQAnXp66KhBVco05tiLKZthkGH9AzwQT33ftf8ErVwT4GXs >>> aEXDdQLctLGxvqfoyys9woNqalYjG9JtZxJHTWfaVU+t3yXEme3GBJBmMlVE >>> GSSk8KAyEil8DP1q4PMQJrScQxqYFpfBi1UGnbiQj02pan16OtbkaUkJNLMB >>> o8XlHdiNfPWmMoyAuOPBMoKSPo1diLBP3uEJN8u3Mw4+9kLZSeMDMRHdkMF0 >>> kmhGA26ihRHcHWVsC+4wevCGJoq7vvPmf8892z+hEjC5vm4eWGAD7UPBjqjl >>> 5BL282XI+AYLbw4VkiDrP4tTL4neOr6IW50mnG8SPVSAvMN+cFJnlMZRpQ/6 >>> SQB4Tv5fr1SEMZDZqC//RacWZYCsBd1XZi6M0VhOhOrGjqmlr/41P2dmrdI1 >>> ldHEyl3l07mJdBANQ0AgIMAeMyuD3dGQ4q0IpJgVMbrfxq8m/lLp/jddOWq3 >>> MhNGT5J1K4Qg3eFlqhuTIw7yLmERACYHlMUBCHGq/8jGCjEQOe6uhuePyH/6 >>> ugUoZ+J4Y+Fxsu1Jsoj+GQtDSSkOTGjUpGhXPp/gqMhxZRlGdsZL5LIEs7hm >>> Us8M >>> =gnRK >>> -----END PGP SIGNATURE----- >> >> -----BEGIN PGP SIGNATURE----- >> Version: Mailvelope v1.2.2 >> Comment: https://www.mailvelope.com >> >> wsFcBAEBCAAQBQJWLaAwCRDmVDuy+mK58QAAAMkP/iJ+OpklHB/O2wgj2Le1 >> Wudniy7KzpDVaS+kXpPZ8Bhbn+rxXCuN9ySUf1sxM37SdqZBCWHpPdx7GTbC >> QDaO2Bucn53iNG6FDcXHf0TDAUuw7f5u381B2+qfuUbc6Q7iJeJRIjrzQsce >> 1ieBDytn+DKis1YEOY5Rlbj80CBB5MhkiokJlxjNjaj2AZJAORwoLbqoCSSI >> u8YnzsbxhkpYxCcCqM3lHf36dsP40vkyXXyqjVWgaW9qThFx9N67ERG/hQSU >> VTBWXqY8glAQmbeuvlT/zAhl0e2qEsEOBUBn5r/ydL5M2x+3dFHsLp92ewwt >> pyrEeq6n8Wt1mmYklesDZQCgex47uWAy5mDFOWQjzWBbeO7ji8jwM+PpXxW9 >> h/mRJZFLLTScFHTOONDXfF41GXFV3ZtdukpHdT46k++RRHmlFVRZgcTym3/b >> g0pLQKZ8ynKvFzAora2/r9IlN7dDPJEw2jpN2pAYda0GlY8wc6h5i/qUQoGE >> VN6b5SNURyw53OMPv6yOx2bvc7RKmpLGWjhnTEHjydI0w+kmbqvAnbT2mG/O >> eHeEyteK4m3+Jtf/s+wN9ULr1pNr++37Zt2igfTtvPm4OceqiU4Y02YgrPu+ >> LgOWGduVSmEmmRRnBE8+gZYU6gSgpOV3JWqP0AQMavujZmQpGs55DjnfMwdv >> v8IJ >> =4TOB >> -----END PGP SIGNATURE----- >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -----BEGIN PGP SIGNATURE----- Version: Mailvelope v1.2.2 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWLjYFCRDmVDuy+mK58QAA6HEP/RDnRwhaXgVp6dfc7k/J 7+ojQXvLJvzAsGo6hzn0faLlq6J6NUII0r5v4uPhnZzy9Sk2F8ro2drJqWKf OXKY/XNWDaVdE6PkA01F8sJvbmE2royaFb4zZdAIWqVxOKy8OHwIR8nRxGA+ 69nqzsMH3pTUoDFPsmDnVIOri7scPiSzno983Nz4A8L1I4JnJOwCFI/H9BhM d6cn5WbV+5zffhwcl8leLiFE97h86C7aZbvj+vC0sCqzijr1Az/rohm8Zt65 KYkvV+7qhFPKGrrb6HahJMV6t3b2bW01yhxiieJ60gOGShSxtHwZa3+ngR4G xG3AFM0NDbQprBTI/kg1BWd84gkt2ww/bzsVtoNRE3vl5qUqTxg6/l1f16jI VumdmFkl5mMNCHb6H3GGUa70lVoRc+ATvEENXALUg35lM2zSU+iDjezJ+Pil boky75J3MVKnOYgac+PGRDe7a9dlWK9eJO7GKnh/GKML3I+DDNM0jSBzZIw8 ld8jy9WXMJe90Am79ZR1rt7hETtPaW4sx7lr19EGKSrsMskAh6suQ/j7npaW 5WHDVtk8hdnrEkjhBXYaHlH/WoTCbCq9r/0Ynpt2dTD32Rb+xHgt1YX7n6sI hagQ5bqT67cGNGCuP7a8tcmNX4To7W9hUhIZcULAuP3OG6R3nOigqt2Y80LV lPih =J9Qy -----END PGP SIGNATURE----- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |