> Op 1 september 2016 om 18:55 schreef Dan Jakubiec <dan.jakubiec@xxxxxxxxx>: > > > Thanks Wido. Reed and I have been working together to try to restore this cluster for about 3 weeks now. I have been accumulating a number of failure modes that I am hoping to share with the Ceph group soon, but have been holding off a bit until we see the full picture clearly so that we can provide some succinct observations. > > We know that losing 6 of 8 OSDs was definitely going to result in data loss, so I think we are resigned to that. What has been difficult for us is that there have been many steps in the rebuild process that seem to get stuck and need our intervention. But it is not 100% obvious what interventions we should applying. > > My very over-simplied hope was this: > > We would remove the corrupted OSDs from the cluster > We would replace them with new OSDs > Ceph would figure out that a lot of PGs were lost > We would "agree and say okay -- lose the objects/files" > The cluster would use what remains and return to working state > > I feel we have done something wrong along the way, and at this point we are trying to figure out how to do step #4 completely. We are about to follow the steps to "mark unfound lost", which makes sense to me... but I'm not sure what to do about all the other inconsistencies. > > What procedure do we need to follow to just tell Ceph "those PGs are lost, let's move on"? > > === > > A very quick history of what we did to get here: > > 8 OSDs lost power simultaneously. > 2 OSDs came back without issues. > 1 OSD wouldn't start (various assertion failures), but we were able to copy its PGs to a new OSD as follows: > ceph-objectstore-tool "export" > ceph osd crush rm osd.N > ceph auth del osd.N > ceph os rm osd.N > Create new OSD from scrach (it got a new OSD ID) > ceph-objectstore-tool "import" > The remaining 5 OSDs were corrupt beyond repair (could not export, mostly due to missing leveldb files after xfs_repair). We redeployed them as follows: > ceph osd crush rm osd.N > ceph auth del osd.N > ceph os rm osd.N > Create new OSD from scratch (it got the same OSD ID as the old OSD) > > All the new OSDs from #4.4 ended up getting the same OSD ID as the original OSD. Don't know if that is part of the problem? It seems like doing the "crush rm" should have advised the cluster correctly, but perhaps not? > > Where did we go wrong in the recovery process? You have to mark those OSDs as lost and also force create the incomplete PGs. But I think you have lost so many objects that the cluster is beyond a point of repair honestly. Wido > > Thank you! > > -- Dan > > > On Sep 1, 2016, at 00:18, Wido den Hollander <wido@xxxxxxxx> wrote: > > > > > >> Op 31 augustus 2016 om 23:21 schreef Reed Dier <reed.dier@xxxxxxxxxxx>: > >> > >> > >> Multiple XFS corruptions, multiple leveldb issues. Looked to be result of write cache settings which have been adjusted now. > >> > > > > That is bad news, really bad. > > > >> You’ll see below that there are tons of PG’s in bad states, and it was slowly but surely bringing the number of bad PGs down, but it seems to have hit a brick wall with this one slow request operation. > >> > > > > No, you have more issues. You can 17 PGs which are incomplete, a few down+incomplete. > > > > Without those PGs functioning (active+X) your MDS will probably not work. > > > > Take a look at: http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/ > > > > Make sure you go to HEALTH_WARN at first, in HEALTH_ERR the MDS will never come online. > > > > Wido > > > >>> ceph -s > >>> cluster [] > >>> health HEALTH_ERR > >>> 292 pgs are stuck inactive for more than 300 seconds > >>> 142 pgs backfill_wait > >>> 135 pgs degraded > >>> 63 pgs down > >>> 80 pgs incomplete > >>> 199 pgs inconsistent > >>> 2 pgs recovering > >>> 5 pgs recovery_wait > >>> 1 pgs repair > >>> 132 pgs stale > >>> 160 pgs stuck inactive > >>> 132 pgs stuck stale > >>> 71 pgs stuck unclean > >>> 128 pgs undersized > >>> 1 requests are blocked > 32 sec > >>> recovery 5301381/46255447 objects degraded (11.461%) > >>> recovery 6335505/46255447 objects misplaced (13.697%) > >>> recovery 131/20781800 unfound (0.001%) > >>> 14943 scrub errors > >>> mds cluster is degraded > >>> monmap e1: 3 mons at {core=[]:6789/0,db=[]:6789/0,dev=[]:6789/0} > >>> election epoch 262, quorum 0,1,2 core,dev,db > >>> fsmap e3627: 1/1/1 up {0=core=up:replay} > >>> osdmap e3685: 8 osds: 8 up, 8 in; 153 remapped pgs > >>> flags sortbitwise > >>> pgmap v1807138: 744 pgs, 10 pools, 7668 GB data, 20294 kobjects > >>> 8998 GB used, 50598 GB / 59596 GB avail > >>> 5301381/46255447 objects degraded (11.461%) > >>> 6335505/46255447 objects misplaced (13.697%) > >>> 131/20781800 unfound (0.001%) > >>> 209 active+clean > >>> 170 active+clean+inconsistent > >>> 112 stale+active+clean > >>> 74 undersized+degraded+remapped+wait_backfill+peered > >>> 63 down+incomplete > >>> 48 active+undersized+degraded+remapped+wait_backfill > >>> 19 stale+active+clean+inconsistent > >>> 17 incomplete > >>> 12 active+remapped+wait_backfill > >>> 5 active+recovery_wait+degraded > >>> 4 undersized+degraded+remapped+inconsistent+wait_backfill+peered > >>> 4 active+remapped+inconsistent+wait_backfill > >>> 2 active+recovering+degraded > >>> 2 undersized+degraded+remapped+peered > >>> 1 stale+active+clean+scrubbing+deep+inconsistent+repair > >>> 1 active+clean+scrubbing+deep > >>> 1 active+clean+scrubbing+inconsistent > >> > >> > >> Thanks, > >> > >> Reed > >> > >>> On Aug 31, 2016, at 4:08 PM, Wido den Hollander <wido@xxxxxxxx> wrote: > >>> > >>>> > >>>> Op 31 augustus 2016 om 22:56 schreef Reed Dier <reed.dier@xxxxxxxxxxx <mailto:reed.dier@xxxxxxxxxxx>>: > >>>> > >>>> > >>>> After a power failure left our jewel cluster crippled, I have hit a sticking point in attempted recovery. > >>>> > >>>> Out of 8 osd’s, we likely lost 5-6, trying to salvage what we can. > >>>> > >>> > >>> That's probably to much. How do you mean lost? Is XFS crippled/corrupted? That shouldn't happen. > >>> > >>>> In addition to rados pools, we were also using CephFS, and the cephfs.metadata and cephfs.data pools likely lost plenty of PG’s. > >>>> > >>> > >>> What is the status of all PGs? What does 'ceph -s' show? > >>> > >>> Are all PGs active? Since that's something which needs to be done first. > >>> > >>>> The mds has reported this ever since returning from the power loss: > >>>>> # ceph mds stat > >>>>> e3627: 1/1/1 up {0=core=up:replay} > >>>> > >>>> > >>>> When looking at the slow request on the osd, it shows this task which I can’t quite figure out. Any help appreciated. > >>>> > >>> > >>> Are all clients (including MDS) and OSDs running the same version? > >>> > >>> Wido > >>> > >>>>> # ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight > >>>>> { > >>>>> "ops": [ > >>>>> { > >>>>> "description": "osd_op(mds.0.3625:8 6.c5265ab3 (undecoded) ack+retry+read+known_if_redirected+full_force e3668)", > >>>>> "initiated_at": "2016-08-31 10:37:18.833644", > >>>>> "age": 22212.235361, > >>>>> "duration": 22212.235379, > >>>>> "type_data": [ > >>>>> "no flag points reached", > >>>>> [ > >>>>> { > >>>>> "time": "2016-08-31 10:37:18.833644", > >>>>> "event": "initiated" > >>>>> } > >>>>> ] > >>>>> ] > >>>>> } > >>>>> ], > >>>>> "num_ops": 1 > >>>>> } > >>>> > >>>> Thanks, > >>>> > >>>> Reed > >>>> _______________________________________________ > >>>> ceph-users mailing list > >>>> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |