Le 14/05/2019 à 10:04, Kevin Flöh a écrit :
On 13.05.19 11:21 nachm., Dan van der Ster wrote:Presumably the 2 OSDs you marked as lost were hosting those incomplete PGs?It would be useful to double confirm that: check with `ceph pg <id> query` and `ceph pg dump`. (If so, this is why the ignore history les thing isn't helping; you don't have the minimum 3 stripes up for those 3+1 PGs.)yes, but as written in my other mail, we still have enough shards, at least I think so.If those "lost" OSDs by some miracle still have the PG data, you might be able to export the relevant PG stripes with the ceph-objectstore-tool. I've never tried this myself, but there have been threads in the past where people export a PG from a nearly dead hdd, import to another OSD, then backfilling works.guess that is not possible.
Hi Kevin, You want to make sure of this.Unless you recreated the OSDs 4 and 23 and had new data written on them, they should still host the data you need. What Dan suggested (export the 7 inconsistent PGs and import them on a healthy OSD) seems to be the only way to recover your lost data, as with 4 hosts and 2 OSDs lost, you're left with 2 chunks of data/parity when you actually need 3 to access it. Reducing min_size to 3 will not help.
Have a look here: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019673.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-January/023736.html This is probably the best way you want to follow form now on. Regards, Frédéric.
If OTOH those PGs are really lost forever, and someone else should confirm what I say here, I think the next step would be to force recreate the incomplete PGs then run a set of cephfs scrub/repair disaster recovery cmds to recover what you can from the cephfs. -- danwould this let us recover at least some of the data on the pgs? If not we would just set up a new ceph directly without fixing the old one and copy whatever is left.Best regards, KevinOn Mon, May 13, 2019 at 4:20 PM Kevin Flöh <kevin.floeh@xxxxxxx> wrote:Dear ceph experts, we have several (maybe related) problems with our ceph cluster, let me first show you the current ceph status: cluster: id: 23e72372-0d44-4cad-b24f-3641b14b86f4 health: HEALTH_ERR 1 MDSs report slow metadata IOs 1 MDSs report slow requests 1 MDSs behind on trimming 1/126319678 objects unfound (0.000%) 19 scrub errorsReduced data availability: 2 pgs inactive, 2 pgs incompletePossible data damage: 7 pgs inconsistent Degraded data redundancy: 1/500333881 objects degraded (0.000%), 1 pg degraded118 stuck requests are blocked > 4096 sec. Implicated osds24,32,91 services: mon: 3 daemons, quorum ceph-node03,ceph-node01,ceph-node02 mgr: ceph-node01(active), standbys: ceph-node01.etp.kit.edu mds: cephfs-1/1/1 up {0=ceph-node02.etp.kit.edu=up:active}, 3 up:standby osd: 96 osds: 96 up, 96 in data: pools: 2 pools, 4096 pgs objects: 126.32M objects, 260TiB usage: 372TiB used, 152TiB / 524TiB avail pgs: 0.049% pgs not active 1/500333881 objects degraded (0.000%) 1/126319678 objects unfound (0.000%) 4076 active+clean 10 active+clean+scrubbing+deep 7 active+clean+inconsistent 2 incomplete 1 active+recovery_wait+degraded io: client: 449KiB/s rd, 42.9KiB/s wr, 152op/s rd, 0op/s wr and ceph health detail:HEALTH_ERR 1 MDSs report slow metadata IOs; 1 MDSs report slow requests;1 MDSs behind on trimming; 1/126319687 objects unfound (0.000%); 19 scrub errors; Reduced data availability: 2 pgs inactive, 2 pgs incomplete; Possible data damage: 7 pgs inconsistent; Degraded data redundancy: 1/500333908 objects degraded (0.000%), 1 pg degraded; 118 stuck requests are blocked > 4096 sec. Implicated osds 24,32,91 MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mdsceph-node02.etp.kit.edu(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 351193 secs MDS_SLOW_REQUEST 1 MDSs report slow requestsmdsceph-node02.etp.kit.edu(mds.0): 4 slow requests are blocked > 30 secMDS_TRIM 1 MDSs behind on trimming mdsceph-node02.etp.kit.edu(mds.0): Behind on trimming (46034/128) max_segments: 128, num_segments: 46034 OBJECT_UNFOUND 1/126319687 objects unfound (0.000%) pg 1.24c has 1 unfound objects OSD_SCRUB_ERRORS 19 scrub errorsPG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs incompletepg 1.5dd is incomplete, acting [24,4,23,79] (reducing pool ec31 min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 1.619 is incomplete, acting [91,23,4,81] (reducing pool ec31 min_size from 3 may help; search ceph.com/docs for 'incomplete') PG_DAMAGED Possible data damage: 7 pgs inconsistent pg 1.17f is active+clean+inconsistent, acting [65,49,25,4] pg 1.1e0 is active+clean+inconsistent, acting [11,32,4,81] pg 1.203 is active+clean+inconsistent, acting [43,49,4,72] pg 1.5d3 is active+clean+inconsistent, acting [37,27,85,4] pg 1.779 is active+clean+inconsistent, acting [50,4,77,62] pg 1.77c is active+clean+inconsistent, acting [21,49,40,4] pg 1.7c3 is active+clean+inconsistent, acting [1,14,68,4] PG_DEGRADED Degraded data redundancy: 1/500333908 objects degraded (0.000%), 1 pg degraded pg 1.24c is active+recovery_wait+degraded, acting [32,4,61,36], 1 unfoundREQUEST_STUCK 118 stuck requests are blocked > 4096 sec. Implicated osds24,32,91 118 ops are blocked > 536871 sec osds 24,32,91 have stuck requests > 536871 sec Let me briefly summarize the setup: We have 4 nodes with 24 osds each and use 3+1 erasure coding. The nodes run on centos7 and we use, due to a major mistake when setting up the cluster, more than one ceph version on the nodes, 3 nodes run on 12.2.12 and one runs on 13.2.5. We are currently not daring to update all nodes to 13.2.5. For all the version details see: { "mon": { "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 3 }, "mgr": { "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 2 }, "osd": { "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 72,"ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988)mimic (stable)": 24 }, "mds": { "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 4 }, "overall": { "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 81,"ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988)mimic (stable)": 24 } }Here is what happened: One osd daemon could not be started and thereforewe decided to mark the osd as lost and set it up from scratch. Ceph started recovering and then we lost another osd with the same behavior.We did the same as for the first osd. And now we are stuck with 2 pgs inincomplete. Ceph pg query gives the following problem: "down_osds_we_would_probe": [], "peering_blocked_by": [], "peering_blocked_by_detail": [ { "detail": "peering_blocked_by_history_les_bound" } We already tried to set "osd_find_best_info_ignore_history_les": "true" for the affected osds, which had no effect. Furthermore, the cluster is behind on trimming by more than 40,000 segments and we have folders and files which cannot be deleted or moved. (which are not on the 2 incomplete pgs). Is there any way to solve these problems? Best regards, Kevin _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Attachment:
smime.p7s
Description: Signature cryptographique S/MIME
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
