Hi,
it sounds like you have auto-repair enabled (osd_scrub_auto_repair). I
guess you could disable that to see what's going on with the PGs and
their replicas. And/or you could enable debug logs. Are all daemons
running the same ceph (minor) version? I remember a customer case
where different ceph minor versions (but overall Octopus) caused
damaged PGs, a repair fixed them everytime. After they updated all
daemons to the same minor version those errors were gone.
Regards,
Eugen
Zitat von Christian Theune <ct@xxxxxxxxxxxxxxx>:
Hi,
this is a bit older cluster (Nautilus, bluestore only).
We’ve noticed that the cluster is almost continuously repairing PGs.
However, they all finish successfully with “0 fixed”. We do not see
the trigger why Ceph decides to repair the PGs and it’s happening
for a lot of PGs, not any specific individual one.
Deep-scrubs are generally running, but currently a bit late as we
had some recoveries in the last week.
Logs look regular aside from the number of repairs. Here’s the last
weeks from the perspective of a single PG. There’s one repair, but
the same thing seems to happen for all PGs.
2023-08-06 16:08:17.870 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-06 16:08:18.270 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-07 21:52:22.299 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-07 21:52:22.711 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-09 00:33:42.587 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-09 00:33:43.049 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-10 09:36:00.590 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub starts
2023-08-10 09:36:28.811 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub ok
2023-08-11 12:59:14.219 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-11 12:59:14.567 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-12 13:52:44.073 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-12 13:52:44.483 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-14 01:51:04.774 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub starts
2023-08-14 01:51:33.113 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub ok
2023-08-15 05:18:16.093 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-15 05:18:16.520 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-16 09:47:38.520 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-16 09:47:38.930 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-17 19:25:45.352 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-17 19:25:45.775 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-19 05:40:43.663 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-19 05:40:44.073 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-20 12:06:54.343 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-20 12:06:54.809 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-21 19:23:10.801 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub starts
2023-08-21 19:23:39.936 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub ok
2023-08-23 03:43:21.391 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-23 03:43:21.844 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-24 04:21:17.004 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub starts
2023-08-24 04:21:47.972 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub ok
2023-08-25 06:55:13.588 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-25 06:55:14.087 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-26 09:26:01.174 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-26 09:26:01.561 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-27 11:18:10.828 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-27 11:18:11.264 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-28 19:05:42.104 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-28 19:05:42.693 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-30 07:03:10.327 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-08-30 07:03:10.805 7fc49f1e6640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-08-31 14:43:23.849 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub starts
2023-08-31 14:43:50.723 7fc49b1de640 0 log_channel(cluster) log
[DBG] : 278.2f3 deep-scrub ok
2023-09-01 20:53:42.749 7f37ca268640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-09-01 20:53:43.389 7f37c6260640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-09-02 22:57:49.542 7f37ca268640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-09-02 22:57:50.065 7f37c6260640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-09-04 03:16:14.754 7f37ca268640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub starts
2023-09-04 03:16:15.295 7f37ca268640 0 log_channel(cluster) log
[DBG] : 278.2f3 scrub ok
2023-09-05 14:50:36.064 7f37ca268640 0 log_channel(cluster) log
[DBG] : 278.2f3 repair starts
2023-09-05 14:51:04.407 7f37c6260640 0 log_channel(cluster) log
[DBG] : 278.2f3 repair ok, 0 fixed
Googling didn’t help, unfortunately and the bug tracker doesn’t
appear to have any relevant issue either.
Any ideas?
Liebe Grüße,
Christian Theune
--
Christian Theune · ct@xxxxxxxxxxxxxxx · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
Zagrodnick
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
