Re: Major ceph disaster

[Kevin Flöh <kevin.floeh@xxxxxxx>]

Hi Frederic,

we do not have access to the original OSDs. We exported the remaining 
shards of the two pgs but we are only left with two shards (of 
reasonable size) per pg. The rest of the shards displayed by ceph pg 
query are empty. I guess marking the OSD as complete doesn't make sense 
then.

Best,
Kevin

On 17.05.19 2:36 nachm., Frédéric Nass wrote:
> Le 14/05/2019 à 10:04, Kevin Flöh a écrit :
> > On 13.05.19 11:21 nachm., Dan van der Ster wrote:
> > > Presumably the 2 OSDs you marked as lost were hosting those 
> > > incomplete PGs?
> > > It would be useful to double confirm that: check with `ceph pg <id>
> > > query` and `ceph pg dump`.
> > > (If so, this is why the ignore history les thing isn't helping; you
> > > don't have the minimum 3 stripes up for those 3+1 PGs.)
> >
> > yes, but as written in my other mail, we still have enough shards, at 
> > least I think so.
> >
> > > If those "lost" OSDs by some miracle still have the PG data, you might
> > > be able to export the relevant PG stripes with the
> > > ceph-objectstore-tool. I've never tried this myself, but there have
> > > been threads in the past where people export a PG from a nearly dead
> > > hdd, import to another OSD, then backfilling works.
> > guess that is not possible.
>
> Hi Kevin,
>
> You want to make sure of this.
>
> Unless you recreated the OSDs 4 and 23 and had new data written on 
> them, they should still host the data you need.
> What Dan suggested (export the 7 inconsistent PGs and import them on a 
> healthy OSD) seems to be the only way to recover your lost data, as 
> with 4 hosts and 2 OSDs lost, you're left with 2 chunks of data/parity 
> when you actually need 3 to access it. Reducing min_size to 3 will not 
> help.
>
> Have a look here:
>
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019673.html
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-January/023736.html 
>
>
> This is probably the best way you want to follow form now on.
>
> Regards,
> Frédéric.
>
> > > If OTOH those PGs are really lost forever, and someone else should
> > > confirm what I say here, I think the next step would be to force
> > > recreate the incomplete PGs then run a set of cephfs scrub/repair
> > > disaster recovery cmds to recover what you can from the cephfs.
> > >
> > > -- dan
> >
> > would this let us recover at least some of the data on the pgs? If 
> > not we would just set up a new ceph directly without fixing the old 
> > one and copy whatever is left.
> >
> > Best regards,
> >
> > Kevin
> >
> >
> >
> > > On Mon, May 13, 2019 at 4:20 PM Kevin Flöh <kevin.floeh@xxxxxxx> wrote:
> > > > Dear ceph experts,
> > > >
> > > > we have several (maybe related) problems with our ceph cluster, let me
> > > > first show you the current ceph status:
> > > >
> > > >     cluster:
> > > >       id:     23e72372-0d44-4cad-b24f-3641b14b86f4
> > > >       health: HEALTH_ERR
> > > >               1 MDSs report slow metadata IOs
> > > >               1 MDSs report slow requests
> > > >               1 MDSs behind on trimming
> > > >               1/126319678 objects unfound (0.000%)
> > > >               19 scrub errors
> > > >               Reduced data availability: 2 pgs inactive, 2 pgs 
> > > > incomplete
> > > >               Possible data damage: 7 pgs inconsistent
> > > >               Degraded data redundancy: 1/500333881 objects degraded
> > > > (0.000%), 1 pg degraded
> > > >               118 stuck requests are blocked > 4096 sec. Implicated 
> > > > osds
> > > > 24,32,91
> > > >
> > > >     services:
> > > >       mon: 3 daemons, quorum ceph-node03,ceph-node01,ceph-node02
> > > >       mgr: ceph-node01(active), standbys: ceph-node01.etp.kit.edu
> > > >       mds: cephfs-1/1/1 up {0=ceph-node02.etp.kit.edu=up:active}, 3
> > > > up:standby
> > > >       osd: 96 osds: 96 up, 96 in
> > > >
> > > >     data:
> > > >       pools:   2 pools, 4096 pgs
> > > >       objects: 126.32M objects, 260TiB
> > > >       usage:   372TiB used, 152TiB / 524TiB avail
> > > >       pgs:     0.049% pgs not active
> > > >                1/500333881 objects degraded (0.000%)
> > > >                1/126319678 objects unfound (0.000%)
> > > >                4076 active+clean
> > > >                10   active+clean+scrubbing+deep
> > > >                7    active+clean+inconsistent
> > > >                2    incomplete
> > > >                1    active+recovery_wait+degraded
> > > >
> > > >     io:
> > > >       client:   449KiB/s rd, 42.9KiB/s wr, 152op/s rd, 0op/s wr
> > > >
> > > >
> > > > and ceph health detail:
> > > >
> > > >
> > > > HEALTH_ERR 1 MDSs report slow metadata IOs; 1 MDSs report slow 
> > > > requests;
> > > > 1 MDSs behind on trimming; 1/126319687 objects unfound (0.000%); 19
> > > > scrub errors; Reduced data availability: 2 pgs inactive, 2 pgs
> > > > incomplete; Possible data damage: 7 pgs inconsistent; Degraded data
> > > > redundancy: 1/500333908 objects degraded (0.000%), 1 pg degraded; 118
> > > > stuck requests are blocked > 4096 sec. Implicated osds 24,32,91
> > > > MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs
> > > >       mdsceph-node02.etp.kit.edu(mds.0): 100+ slow metadata IOs are
> > > > blocked > 30 secs, oldest blocked for 351193 secs
> > > > MDS_SLOW_REQUEST 1 MDSs report slow requests
> > > >       mdsceph-node02.etp.kit.edu(mds.0): 4 slow requests are 
> > > > blocked > 30 sec
> > > > MDS_TRIM 1 MDSs behind on trimming
> > > >       mdsceph-node02.etp.kit.edu(mds.0): Behind on trimming 
> > > > (46034/128)
> > > > max_segments: 128, num_segments: 46034
> > > > OBJECT_UNFOUND 1/126319687 objects unfound (0.000%)
> > > >       pg 1.24c has 1 unfound objects
> > > > OSD_SCRUB_ERRORS 19 scrub errors
> > > > PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs 
> > > > incomplete
> > > >       pg 1.5dd is incomplete, acting [24,4,23,79] (reducing pool ec31
> > > > min_size from 3 may help; search ceph.com/docs for 'incomplete')
> > > >       pg 1.619 is incomplete, acting [91,23,4,81] (reducing pool ec31
> > > > min_size from 3 may help; search ceph.com/docs for 'incomplete')
> > > > PG_DAMAGED Possible data damage: 7 pgs inconsistent
> > > >       pg 1.17f is active+clean+inconsistent, acting [65,49,25,4]
> > > >       pg 1.1e0 is active+clean+inconsistent, acting [11,32,4,81]
> > > >       pg 1.203 is active+clean+inconsistent, acting [43,49,4,72]
> > > >       pg 1.5d3 is active+clean+inconsistent, acting [37,27,85,4]
> > > >       pg 1.779 is active+clean+inconsistent, acting [50,4,77,62]
> > > >       pg 1.77c is active+clean+inconsistent, acting [21,49,40,4]
> > > >       pg 1.7c3 is active+clean+inconsistent, acting [1,14,68,4]
> > > > PG_DEGRADED Degraded data redundancy: 1/500333908 objects degraded
> > > > (0.000%), 1 pg degraded
> > > >       pg 1.24c is active+recovery_wait+degraded, acting 
> > > > [32,4,61,36], 1
> > > > unfound
> > > > REQUEST_STUCK 118 stuck requests are blocked > 4096 sec. Implicated 
> > > > osds
> > > > 24,32,91
> > > >       118 ops are blocked > 536871 sec
> > > >       osds 24,32,91 have stuck requests > 536871 sec
> > > >
> > > >
> > > > Let me briefly summarize the setup: We have 4 nodes with 24 osds each
> > > > and use 3+1 erasure coding. The nodes run on centos7 and we use, 
> > > > due to
> > > > a major mistake when setting up the cluster, more than one ceph 
> > > > version
> > > > on the nodes, 3 nodes run on 12.2.12 and one runs on 13.2.5. We are
> > > > currently not daring to update all nodes to 13.2.5. For all the 
> > > > version
> > > > details see:
> > > >
> > > > {
> > > >       "mon": {
> > > >           "ceph version 12.2.12
> > > > (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 3
> > > >       },
> > > >       "mgr": {
> > > >           "ceph version 12.2.12
> > > > (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 2
> > > >       },
> > > >       "osd": {
> > > >           "ceph version 12.2.12
> > > > (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 72,
> > > >           "ceph version 13.2.5 
> > > > (cbff874f9007f1869bfd3821b7e33b2a6ffd4988)
> > > > mimic (stable)": 24
> > > >       },
> > > >       "mds": {
> > > >           "ceph version 12.2.12
> > > > (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 4
> > > >       },
> > > >       "overall": {
> > > >           "ceph version 12.2.12
> > > > (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 81,
> > > >           "ceph version 13.2.5 
> > > > (cbff874f9007f1869bfd3821b7e33b2a6ffd4988)
> > > > mimic (stable)": 24
> > > >       }
> > > > }
> > > >
> > > > Here is what happened: One osd daemon could not be started and 
> > > > therefore
> > > > we decided to mark the osd as lost and set it up from scratch. Ceph
> > > > started recovering and then we lost another osd with the same 
> > > > behavior.
> > > > We did the same as for the first osd. And now we are stuck with 2 
> > > > pgs in
> > > > incomplete. Ceph pg query gives the following problem:
> > > >
> > > >               "down_osds_we_would_probe": [],
> > > >               "peering_blocked_by": [],
> > > >               "peering_blocked_by_detail": [
> > > >                   {
> > > >                       "detail": "peering_blocked_by_history_les_bound"
> > > >                   }
> > > >
> > > > We already tried to set "osd_find_best_info_ignore_history_les": 
> > > > "true"
> > > > for the affected osds, which had no effect. Furthermore, the 
> > > > cluster is
> > > > behind on trimming by more than 40,000 segments and we have folders 
> > > > and
> > > > files which cannot be deleted or moved. (which are not on the 2
> > > > incomplete pgs). Is there any way to solve these problems?
> > > >
> > > > Best regards,
> > > >
> > > > Kevin
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users@xxxxxxxxxxxxxx
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com