Hi Wesley,
On 06/10/2023 17:48, Wesley Dillingham wrote:
A repair is just a type of scrub and it is also limited by
osd_max_scrubs which in pacific is 1.
We've increased that to 4 (and temporarily to 8) since we have so many
OSDs and are running behind on scrubbing.
If another scrub is occurring on any OSD in the PG it wont start.
that explains a lot.
do "ceph osd set noscrub" and "ceph osd set nodeep-scrub" wait for all
scrubs to stop (a few seconds probably)
Then issue the pg repair command again. It may start.
The script Kai linked seems like a good idea to fix this when needed.
You also have pgs in backfilling state. Note that by default OSDs in
backfill or backfill_wait also wont perform scrubs.
You can modify this behavior with `ceph config set osd
osd_scrub_during_recovery true`
We've set this already
I would suggest only setting that after the noscub flags are set and the
only scrub you want to get processed is your manual repair.
Then rm the scrub_during_recovery config item before unsetting the
noscrub flags.
Thanks for the suggestion!
Cheers
/Simon
Respectfully,
*Wes Dillingham*
wes@xxxxxxxxxxxxxxxxx <mailto:wes@xxxxxxxxxxxxxxxxx>
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Fri, Oct 6, 2023 at 11:02 AM Simon Oosthoek <s.oosthoek@xxxxxxxxxxxxx
<mailto:s.oosthoek@xxxxxxxxxxxxx>> wrote:
On 06/10/2023 16:09, Simon Oosthoek wrote:
> Hi
>
> we're still in HEALTH_ERR state with our cluster, this is the top
of the
> output of `ceph health detail`
>
> HEALTH_ERR 1/846829349 objects unfound (0.000%); 248 scrub errors;
> Possible data damage: 1 pg recovery_unfound, 2 pgs inconsistent;
> Degraded data redundancy: 6/7118781559 objects degraded (0.000%),
1 pg
> degraded, 1 pg undersized; 63 pgs not deep-scrubbed in time; 657
pgs not
> scrubbed in time
> [WRN] OBJECT_UNFOUND: 1/846829349 objects unfound (0.000%)
> pg 26.323 has 1 unfound objects
> [ERR] OSD_SCRUB_ERRORS: 248 scrub errors
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 2 pgs
> inconsistent
> pg 26.323 is active+recovery_unfound+degraded+remapped, acting
> [92,109,116,70,158,128,243,189,256], 1 unfound
> pg 26.337 is active+clean+inconsistent, acting
> [139,137,48,126,165,89,237,199,189]
> pg 26.3e2 is active+clean+inconsistent, acting
> [12,27,24,234,195,173,98,32,35]
> [WRN] PG_DEGRADED: Degraded data redundancy: 6/7118781559 objects
> degraded (0.000%), 1 pg degraded, 1 pg undersized
> pg 13.3a5 is stuck undersized for 4m, current state
> active+undersized+remapped+backfilling, last acting
> [2,45,32,62,2147483647,55,116,25,225,202,240]
> pg 26.323 is active+recovery_unfound+degraded+remapped, acting
> [92,109,116,70,158,128,243,189,256], 1 unfound
>
>
> For the PG_DAMAGED pgs I try the usual `ceph pg repair 26.323` etc.,
> however it fails to get resolved.
>
> The osd.116 is already marked out and is beginning to get empty.
I've
> tried restarting the osd processes of the first osd listed for
each PG,
> but that doesn't get it resolved either.
>
> I guess we should have enough redundancy to get the correct data
back,
> but how can I tell ceph to fix it in order to get back to a
healthy state?
I guess this could be related to the number of scrubs going on, I read
somewhere that this may interfere with the repair request. I would
expect the repair would have priority over scrubs...
BTW, we're running pacific for now, we want to update when the cluster
is healthy again.
Cheers
/Simon
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
