On Thu, Jun 30, 2016 at 3:22 AM, Brian Felton <bjfelton@xxxxxxxxx> wrote: > Greetings, > > I have a lab cluster running Hammer 0.94.6 and being used exclusively for > object storage. The cluster consists of four servers running 60 6TB OSDs > each. The main .rgw.buckets pool is using k=3 m=1 erasure coding and > contains 8192 placement groups. > > Last week, one of our guys out-ed and removed one OSD from each of three of > the four servers in the cluster, which resulted in some general badness (the > disks were wiped post-removal, so the data are gone). After a proper > education in why this is a Bad Thing, we got the OSDs added back. When all > was said and done, we had 30 pgs that were stuck incomplete, and no amount > of magic has been able to get them to recover. From reviewing the data, we > knew that all of these pgs contained at least 2 of the removed OSDs; I > understand and accept that the data are gone, and that's not a concern (yay > lab). > > Here are the things I've tried: > > - Restarted all OSDs > - Stopped all OSDs, removed all OSDs from the crush map, and started > everything back up > - Executed a 'ceph pg force_create_pg <id>' for each of the 30 stuck pgs > - Executed a 'ceph pg send_pg_creates' to get the ball rolling on creates > - Executed several 'ceph pg <id> query' commands to ensure we were > referencing valid OSDs after the 'force_create_pg' > - Ensured those OSDs were really removed (e.g. 'ceph auth del', 'ceph osd > crush remove', and 'ceph osd rm') Can you share some of the pg query output? > > At this point, I've got the same 30 pgs that are stuck creating. I've run > out of ideas for getting this back to a healthy state. In reviewing the > other posts on the mailing list, the overwhelming solution was a bad OSD in > the crush map, but I'm all but certain that isn't what's hitting us here. > Normally, being the lab, I'd consider nuking the .rgw.buckets pool and > starting from scratch, but we've recently spent a lot of time pulling 140TB > of data into this cluster for some performance and recovery tests, and I'd > prefer not to have to start that process again. I am willing to entertain > most any other idea irrespective to how destructive it is to these PGs, so > long as I don't have to lose the rest of the data in the pool. > > Many thanks in advance for any assistance here. > > Brian Felton > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |