Le 20/09/2024 à 08:00:16+0000, Eugen Block a écrit
Hi,
>
> there's some ratio involved when deep-scrubs are checked:
>
> (mon_warn_pg_not_deep_scrubbed_ratio * deep_scrub_interval) +
> deep_scrub_interval
>
> So based on the defaults, ceph would only warn if the last deep-scrub
> timestamp is older than:
>
> (0.75 * 7 days) + 7 days = 12.25 days
OK. Thanks. It fit perfectly.
>
> > Is they are any way to find which pg ceph status are talking about.
>
> 'ceph health detail' will show you which PG it's warning about.
Too easy for me ;-) ;-)...Thanks ;-)
>
> > Is they are any way to see the progress or scrubbing/remapping/backfill ?
>
> You can see when (deep-)scrubs have been started in the OSD logs or
> depending on your cluster log configuration:
>
> ceph log last 1000 debug cluster | grep scrub
Thanks.
So I think I get some issue with one osd
2024-09-20T08:55:59.760766+0000 osd.356 (osd.356) 84 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:03.751487+0000 osd.356 (osd.356) 85 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:05.742816+0000 osd.356 (osd.356) 86 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:07.822277+0000 osd.356 (osd.356) 87 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:08.795748+0000 osd.356 (osd.356) 88 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:09.749838+0000 osd.356 (osd.356) 89 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:10.778235+0000 osd.356 (osd.356) 90 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:14.792102+0000 osd.356 (osd.356) 91 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:15.832620+0000 osd.356 (osd.356) 92 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:16.791811+0000 osd.356 (osd.356) 93 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:17.798181+0000 osd.356 (osd.356) 94 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:19.793526+0000 osd.356 (osd.356) 95 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:20.809140+0000 osd.356 (osd.356) 96 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:21.835052+0000 osd.356 (osd.356) 97 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:22.817378+0000 osd.356 (osd.356) 98 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:28.887092+0000 osd.356 (osd.356) 99 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:32.907468+0000 osd.356 (osd.356) 100 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:33.900065+0000 osd.356 (osd.356) 101 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:37.853769+0000 osd.356 (osd.356) 102 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:38.850513+0000 osd.356 (osd.356) 103 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:46.799896+0000 osd.356 (osd.356) 104 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:50.915450+0000 osd.356 (osd.356) 105 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:53.875875+0000 osd.356 (osd.356) 106 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:55.791675+0000 osd.356 (osd.356) 107 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:57.833012+0000 osd.356 (osd.356) 108 : cluster [DBG] 4.51 deep-scrub starts
2024-09-20T08:56:59.818262+0000 osd.356 (osd.356) 109 : cluster [DBG] 4.51 deep-scrub starts
but the osd are still in
queued for deep scrub
in ceph pg dump pgs
>
> The (deep-)scrub duration depends on the PG sizes, so they can vary. But
> from experience (and older Logs) you can see if scrubbing duration has
> increased. I haven't checked if there's a metric for that in prometheus.
> As for remapping and backfill operations, they are constantly reported in
> 'ceph status', it shows how many objects are degraded, how many PGs are
> remapped etc. If you mean something else, please clarify.
>
Yes. But the point is I got
7 active+remapped+backfilling
2 active+remapped+backfill_wait
but I've no idea if that's the 7 (and 2 )
same pg and are stuck in the process since X days.
>
> [0] https://docs.ceph.com/en/latest/rados/operations/health-checks/#pg-not-deep-scrubbed
> [1] https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/
Thanks...
Regards.
--
Albert SHIH 🦫 🐸
Observatoire de Paris
France
Heure locale/Local time:
ven. 20 sept. 2024 10:55:41 CEST
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
