Of course I found the cause shortly after sending the message … The scrubbing parameters need to move from the [osd] section to the [global] section, see https://www.suse.com/support/kb/doc/?id=000019621 Health is back to OK after restarting osds, mons and mgrs. Cheers, Dirk On Dienstag, 11. August 2020 09:34:36 CEST Dirk Sarpe wrote: > Hi, > > since some time (I think upgrade to nautilus) we get > > X pgs not deep scrubbed in time > > I deep-scrubbed the pgs when the error occurred and expected the cluster to > recover over time, but no such luck. The warning comes up again and again. > > In our spinning rust cluster we allow deep scrubbing only from 19:00 to > 06:00 and changed the deep scrub interval to 28 days (detailed osd scrub > config below). Looking at the pgs which are supposedly not deep scrubbed in > time, reveals that their 28 day period is not over yet, example: > > # date && ceph health detail | awk '$1=="pg" { print $0 }' > Di 11. Aug 09:32:17 CEST 2020 > pg 0.787 not deep-scrubbed since 2020-07-30 03:08:24.899264 > pg 0.70c not deep-scrubbed since 2020-07-30 02:45:08.989329 > pg 0.6c1 not deep-scrubbed since 2020-07-30 03:01:15.199496 > pg 13.3 not deep-scrubbed since 2020-07-30 03:29:54.536825 > pg 0.d9 not deep-scrubbed since 2020-07-30 03:12:34.503586 > pg 0.41a not deep-scrubbed since 2020-07-30 03:01:23.514582 > pg 0.490 not deep-scrubbed since 2020-07-30 03:05:45.616100 > > > I wonder if I have missed or messed up some parameter that could cause this. > > # ceph daemon osd.40 config show | grep scrub > "mds_max_scrub_ops_in_progress": "5", > "mon_scrub_inject_crc_mismatch": "0.000000", > "mon_scrub_inject_missing_keys": "0.000000", > "mon_scrub_interval": "86400", > "mon_scrub_max_keys": "100", > "mon_scrub_timeout": "300", > "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", > "mon_warn_pg_not_scrubbed_ratio": "0.500000", > "osd_debug_deep_scrub_sleep": "0.000000", > "osd_deep_scrub_interval": "2419200.000000", > "osd_deep_scrub_keys": "1024", > "osd_deep_scrub_large_omap_object_key_threshold": "200000", > "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", > "osd_deep_scrub_randomize_ratio": "0.150000", > "osd_deep_scrub_stride": "1048576", > "osd_deep_scrub_update_digest_min_age": "7200", > "osd_max_scrubs": "1", > "osd_op_queue_mclock_scrub_lim": "0.001000", > "osd_op_queue_mclock_scrub_res": "0.000000", > "osd_op_queue_mclock_scrub_wgt": "1.000000", > "osd_requested_scrub_priority": "120", > "osd_scrub_auto_repair": "false", > "osd_scrub_auto_repair_num_errors": "5", > "osd_scrub_backoff_ratio": "0.660000", > "osd_scrub_begin_hour": "19", > "osd_scrub_begin_week_day": "0", > "osd_scrub_chunk_max": "25", > "osd_scrub_chunk_min": "5", > "osd_scrub_cost": "52428800", > "osd_scrub_during_recovery": "false", > "osd_scrub_end_hour": "6", > "osd_scrub_end_week_day": "7", > "osd_scrub_interval_randomize_ratio": "0.500000", > "osd_scrub_invalid_stats": "true", > "osd_scrub_load_threshold": "0.500000", > "osd_scrub_max_interval": "604800.000000", > "osd_scrub_max_preemptions": "5", > "osd_scrub_min_interval": "172800.000000", > "osd_scrub_priority": "5", > "osd_scrub_sleep": "0.100000", > > > ceph version 14.2.10 > > My ad hoc helper script > > #!/bin/bash > > # gently deep scrub all pgs which failed to deep scrub in set period > > ceph health detail | > awk '$1=="pg" { print $2 } ' | > while read -r pg > do > ceph pg deep-scrub "$pg" > sleep 60 > done > > > Cheers, > Dirk > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx -- general it-support unit Phone +49 341 97-33118 Email dirk.sarpe@xxxxxxx <mailto:dirk.sarpe@xxxxxxx> German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig Deutscher Platz 5e 04103 Leipzig Germany iDiv is a research centre of the DFG - Deutsche Forschungsgemeinschaft iDiv ist ein Forschungszentrum der Deutschen Forschungsgemeinschaft (DFG). Es ist eine zentrale Einrichtung der Universität Leipzig im Sinne des § 92 Abs. 1 SächsHSFG und wird zusammen mit der Martin-Luther-Universität Halle-Wittenberg, der Friedrich-Schiller-Universität Jena sowie dem Helmholtz-Zentrum für Umweltforschung (UFZ) betrieben. Sieben außeruniversitäre Einrichtungen unterstützen iDiv finanziell sowie durch ihre Expertise: das Max-Planck-Institut für Biogeochemie (MPI BGC), das Max-Planck-Institut für chemische Ökologie (MPI CE), das Max-Planck-Institut für evolutionäre Anthropologie (MPI EVA), das Leibniz-Institut Deutsche Sammlung von Mikroorganismen und Zellkulturen (DSMZ), das Leibniz-Institut für Pflanzenbiochemie (IPB), das Leibnitz-Institut für Pflanzengenetik und Kulturpflanzenforschung (IPK) und das Leibniz-Institut Senckenberg Museum für Naturkunde Görlitz (SMNG). USt-IdNr. DE 141510383 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |