Hi, this is a bit older cluster (Nautilus, bluestore only). We’ve noticed that the cluster is almost continuously repairing PGs. However, they all finish successfully with “0 fixed”. We do not see the trigger why Ceph decides to repair the PGs and it’s happening for a lot of PGs, not any specific individual one. Deep-scrubs are generally running, but currently a bit late as we had some recoveries in the last week. Logs look regular aside from the number of repairs. Here’s the last weeks from the perspective of a single PG. There’s one repair, but the same thing seems to happen for all PGs. 2023-08-06 16:08:17.870 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-06 16:08:18.270 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-07 21:52:22.299 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-07 21:52:22.711 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-09 00:33:42.587 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-09 00:33:43.049 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-10 09:36:00.590 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-10 09:36:28.811 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-11 12:59:14.219 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-11 12:59:14.567 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-12 13:52:44.073 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-12 13:52:44.483 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-14 01:51:04.774 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-14 01:51:33.113 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-15 05:18:16.093 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-15 05:18:16.520 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-16 09:47:38.520 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-16 09:47:38.930 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-17 19:25:45.352 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-17 19:25:45.775 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-19 05:40:43.663 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-19 05:40:44.073 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-20 12:06:54.343 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-20 12:06:54.809 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-21 19:23:10.801 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-21 19:23:39.936 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-23 03:43:21.391 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-23 03:43:21.844 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-24 04:21:17.004 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-24 04:21:47.972 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-25 06:55:13.588 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-25 06:55:14.087 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-26 09:26:01.174 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-26 09:26:01.561 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-27 11:18:10.828 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-27 11:18:11.264 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-28 19:05:42.104 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-28 19:05:42.693 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-30 07:03:10.327 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-30 07:03:10.805 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-31 14:43:23.849 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-31 14:43:50.723 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-09-01 20:53:42.749 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-01 20:53:43.389 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-02 22:57:49.542 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-02 22:57:50.065 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-04 03:16:14.754 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-04 03:16:15.295 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-05 14:50:36.064 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 repair starts 2023-09-05 14:51:04.407 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 repair ok, 0 fixed Googling didn’t help, unfortunately and the bug tracker doesn’t appear to have any relevant issue either. Any ideas? Liebe Grüße, Christian Theune -- Christian Theune · ct@xxxxxxxxxxxxxxx · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |