Hi Eugen, thanks for your answer. I gave a search another try and did indeed find something: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/TN6WJVCHTVJ4YIA4JH2D2WYYZFZRMSXI/ Quote: " ... And I've also observed that the repair req isn't queued up -- if the OSDs are busy with other scrubs, the repair req is forgotten. ..." I'm biting my tongue really really hard right now. @Dan (if you read this), thanks for the script: https://github.com/cernceph/ceph-scripts/blob/master/tools/scrubbing/autorepair.sh New status: # ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 8d) mgr: ceph-25(active, since 8d), standbys: ceph-26, ceph-03, ceph-02, ceph-01 mds: con-fs2:8 4 up:standby 8 up:active osd: 1086 osds: 1071 up (since 14h), 1070 in (since 4d); 542 remapped pgs task status: data: pools: 14 pools, 17185 pgs objects: 1.39G objects, 2.5 PiB usage: 3.1 PiB used, 8.4 PiB / 11 PiB avail pgs: 301878494/11947144857 objects misplaced (2.527%) 16634 active+clean 513 active+remapped+backfill_wait 19 active+remapped+backfilling 10 active+remapped+backfill_wait+forced_backfill 6 active+clean+scrubbing+deep 2 active+clean+scrubbing 1 active+clean+scrubbing+deep+inconsistent+repair io: client: 444 MiB/s rd, 446 MiB/s wr, 2.19k op/s rd, 2.34k op/s wr recovery: 0 B/s, 223 objects/s Yay! Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@xxxxxx> Sent: 13 October 2022 23:23:10 To: ceph-users@xxxxxxx Subject: Re: pg repair doesn't start Hi, I’m not sure if I remember correctly but I believe the backfill is preventing the repair to happen. I think it has been discussed a couple of times on this list but I don’t know right now if you can tweak anything to prioritize the repair, I believe there is, but not sure. It looks like your backfill could take quite some time… Zitat von Frank Schilder <frans@xxxxxx>: > Hi all, > > we have an inconsistent PG for a couple of days now (octopus latest): > > # ceph status > cluster: > id: > health: HEALTH_ERR > 1 scrub errors > Possible data damage: 1 pg inconsistent > > services: > mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 8d) > mgr: ceph-25(active, since 8d), standbys: ceph-26, ceph-03, > ceph-02, ceph-01 > mds: con-fs2:8 4 up:standby 8 up:active > osd: 1086 osds: 1071 up (since 13h), 1070 in (since 4d); 547 remapped pgs > > task status: > > data: > pools: 14 pools, 17185 pgs > objects: 1.39G objects, 2.5 PiB > usage: 3.1 PiB used, 8.4 PiB / 11 PiB avail > pgs: 305530535/11943726075 objects misplaced (2.558%) > 16614 active+clean > 516 active+remapped+backfill_wait > 23 active+clean+scrubbing+deep > 21 active+remapped+backfilling > 10 active+remapped+backfill_wait+forced_backfill > 1 active+clean+inconsistent > > io: > client: 143 MiB/s rd, 135 MiB/s wr, 2.21k op/s rd, 2.33k op/s wr > recovery: 0 B/s, 224 objects/s > > I issued "ceph pg repair 11.1ba" more than 36 hours ago, but it > never got executed (checked the logs for repair state). The usual > wait time we had on our cluster so far was 2-6 hours. 36 hours is > unusually long. The pool in question is moderately busy and has no > misplaced ojects. Its only unhealthy PG is the inconsistent one. > > Are there situations in which ceph cancels/ignores a pg repair? > Is there any way to check if it is actually still scheduled to happen? > Is there a way to force it a bit more urgently? > > The error was caused by a read error, the drive is healthy: > > 2022-10-11T19:19:13.621470+0200 osd.231 (osd.231) 40 : cluster [ERR] > 11.1ba shard 294(6) soid > 11:5df75341:::rbd_data.1.b688997dc79def.000000000005d530:head : > candidate had a read error > 2022-10-11T19:26:22.344862+0200 osd.231 (osd.231) 41 : cluster [ERR] > 11.1bas0 deep-scrub 0 missing, 1 inconsistent objects > 2022-10-11T19:26:22.344866+0200 osd.231 (osd.231) 42 : cluster [ERR] > 11.1ba deep-scrub 1 errors > 2022-10-11T19:26:23.356402+0200 mgr.ceph-25 (mgr.144330518) 378551 : > cluster [DBG] pgmap v301249: 17334 pgs: 1 active+clean+inconsistent, > 2 active+clean+scrubbing, 26 active+remapped+backfill_wait, 13 > active+remapped+backfilling, 19 active+clean+scrubbing+deep, 17273 > active+clean; 2.5 PiB data, 3.1 PiB used, 8.4 PiB / 11 PiB avail; > 193 MiB/s rd, 181 MiB/s wr, 4.95k op/s; 16126995/11848511097 objects > misplaced (0.136%); 0 B/s, 513 objects/s recovering > 2022-10-11T19:26:24.246194+0200 mon.ceph-01 (mon.0) 633486 : cluster > [ERR] Health check failed: 1 scrub errors (OSD_SCRUB_ERRORS) > 2022-10-11T19:26:24.246215+0200 mon.ceph-01 (mon.0) 633487 : cluster > [ERR] Health check failed: Possible data damage: 1 pg inconsistent > (PG_DAMAGED) > > Thanks and best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |