I was running into that as well. Setting `osd_mclock_override_recovery_settings` [1] to true allowed me to manage osd_max_backfills again and get recovery to start happening again. It's on my todo list to understand mclock profiles, but resizing PGs was a nightmare with it. Changing to override the recovery settings saved me. [1] https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/#confval-osd_mclock_override_recovery_settings On Fri, Oct 25, 2024 at 11:13 AM Joffrey <joff.au@xxxxxxxxx> wrote: > HI, > > > This is my cluster: > > cluster: > id: c300532c-51fa-11ec-9a41-0050569c3b55 > health: HEALTH_WARN > Degraded data redundancy: 2062374/1331064781 objects degraded > (0.155%), 278 pgs degraded, 40 pgs undersized > 2497 pgs not deep-scrubbed in time > 2497 pgs not scrubbed in time > > services: > mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 > (age 9d) > mgr: hbgt-ceph1-mon3.gmfzqm(active, since 10d), standbys: > hbgt-ceph1-mon2.nteihj, hbgt-ceph1-mon1.thrnnu > osd: 96 osds: 96 up (since 9d), 96 in (since 45h); 1588 remapped pgs > rgw: 3 daemons active (3 hosts, 2 zones) > > data: > pools: 16 pools, 2497 pgs > objects: 266.22M objects, 518 TiB > usage: 976 TiB used, 808 TiB / 1.7 PiB avail > pgs: 2062374/1331064781 objects degraded (0.155%) > 349917519/1331064781 objects misplaced (26.289%) > 1312 active+remapped+backfill_wait > 864 active+clean > 199 active+recovery_wait+degraded+remapped > 38 active+recovery_wait+degraded > 33 active+undersized+degraded+remapped+backfill_wait > 33 active+recovery_wait+remapped > 7 active+recovery_wait > 6 active+undersized+degraded+remapped+backfilling > 2 active+recovering+remapped > 1 active+remapped+backfilling > 1 active+recovering+degraded+remapped > 1 active+recovery_wait+undersized+degraded+remapped > > io: > client: 683 KiB/s rd, 2.2 KiB/s wr, 51 op/s rd, 2 op/s wr > > > No recovery is running and I don't understand why. > I have free space: > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS TYPE NAME > -1 1784.12231 - 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 > TiB 808 TiB 54.72 1.00 - root default > -5 208.09680 - 208 TiB 142 TiB 130 TiB 51 GiB 605 > GiB 66 TiB 68.14 1.25 - host hbgt-ceph1-osd01 > 1 hdd 17.34140 1.00000 17 TiB 11 TiB 11 TiB 33 KiB 49 > GiB 5.9 TiB 66.16 1.21 136 up osd.1 > 3 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 23 GiB 49 > GiB 6.3 TiB 63.80 1.17 139 up osd.3 > 5 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 139 MiB 53 > GiB 4.8 TiB 72.31 1.32 142 up osd.5 > 7 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 51 > GiB 5.6 TiB 67.97 1.24 145 up osd.7 > 9 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 2.2 GiB 49 > GiB 6.0 TiB 65.67 1.20 140 up osd.9 > 11 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 329 MiB 50 > GiB 5.5 TiB 68.42 1.25 145 up osd.11 > 13 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 > GiB 5.1 TiB 70.45 1.29 153 up osd.13 > 15 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 61 KiB 48 > GiB 5.7 TiB 66.85 1.22 144 up osd.15 > 17 hdd 17.34140 1.00000 17 TiB 11 TiB 9.5 TiB 272 MiB 45 > GiB 6.8 TiB 60.63 1.11 120 up osd.17 > 19 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 12 GiB 50 > GiB 5.9 TiB 65.90 1.20 134 up osd.19 > 21 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.6 GiB 57 > GiB 4.1 TiB 76.49 1.40 152 up osd.21 > 23 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 31 KiB 54 > GiB 4.7 TiB 73.10 1.34 124 up osd.23 > -3 208.09680 - 208 TiB 146 TiB 134 TiB 64 GiB 629 > GiB 62 TiB 70.05 1.28 - host hbgt-ceph1-osd02 > 0 hdd 17.34140 1.00000 17 TiB 11 TiB 9.8 TiB 22 GiB 49 > GiB 6.6 TiB 62.07 1.13 124 up osd.0 > 2 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.7 GiB 52 > GiB 5.2 TiB 70.14 1.28 150 up osd.2 > 4 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 48 > GiB 5.8 TiB 66.83 1.22 152 up osd.4 > 6 hdd 17.34140 0.85004 17 TiB 13 TiB 12 TiB 11 GiB 58 > GiB 4.0 TiB 76.85 1.40 153 up osd.6 > 8 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 54 > GiB 4.9 TiB 71.58 1.31 152 up osd.8 > 10 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 6.3 MiB 47 > GiB 6.1 TiB 64.91 1.19 133 up osd.10 > 12 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 109 MiB 51 > GiB 5.6 TiB 67.72 1.24 137 up osd.12 > 14 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 GiB 53 > GiB 5.1 TiB 70.37 1.29 148 up osd.14 > 16 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 2.9 GiB 54 > GiB 4.9 TiB 71.65 1.31 145 up osd.16 > 18 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 88 MiB 55 > GiB 4.8 TiB 72.45 1.32 154 up osd.18 > 20 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 108 MiB 55 > GiB 4.6 TiB 73.39 1.34 166 up osd.20 > 22 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 209 MiB 54 > GiB 4.7 TiB 72.68 1.33 138 up osd.22 > -7 208.09680 - 208 TiB 148 TiB 136 TiB 29 GiB 635 > GiB 60 TiB 71.03 1.30 - host hbgt-ceph1-osd03 > 24 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.4 GiB 56 > GiB 4.5 TiB 74.27 1.36 150 up osd.24 > 25 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 32 KiB 57 > GiB 4.1 TiB 76.27 1.39 162 up osd.25 > 26 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 1.4 GiB 49 > GiB 6.0 TiB 65.30 1.19 134 up osd.26 > 27 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.7 GiB 53 > GiB 4.7 TiB 73.16 1.34 152 up osd.27 > 28 hdd 17.34140 1.00000 17 TiB 12 TiB 12 TiB 2.3 MiB 52 > GiB 4.9 TiB 71.97 1.32 158 up osd.28 > 29 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 10 GiB 52 > GiB 5.3 TiB 69.72 1.27 141 up osd.29 > 30 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 266 MiB 54 > GiB 4.6 TiB 73.38 1.34 142 up osd.30 > 31 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 93 MiB 50 > GiB 5.7 TiB 67.36 1.23 145 up osd.31 > 32 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 13 GiB 50 > GiB 6.1 TiB 64.77 1.18 131 up osd.32 > 33 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 MiB 53 > GiB 5.2 TiB 69.77 1.28 132 up osd.33 > 34 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 242 MiB 50 > GiB 5.5 TiB 68.17 1.25 139 up osd.34 > 35 hdd 17.34140 1.00000 17 TiB 14 TiB 13 TiB 135 MiB 58 > GiB 3.8 TiB 78.26 1.43 157 up osd.35 > -9 208.09680 - 208 TiB 146 TiB 134 TiB 38 GiB 627 > GiB 62 TiB 70.19 1.28 - host hbgt-ceph1-osd04 > 36 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 210 MiB 47 > GiB 6.4 TiB 63.16 1.15 139 up osd.36 > 37 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 222 MiB 53 > GiB 5.0 TiB 71.34 1.30 147 up osd.37 > 38 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 13 MiB 55 > GiB 4.5 TiB 74.33 1.36 155 up osd.38 > 39 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 135 MiB 52 > GiB 5.2 TiB 70.22 1.28 140 up osd.39 > 40 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 25 MiB 53 > GiB 4.9 TiB 71.46 1.31 150 up osd.40 > 41 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 54 > GiB 5.0 TiB 71.36 1.30 150 up osd.41 > 42 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 823 MiB 50 > GiB 5.8 TiB 66.49 1.22 156 up osd.42 > 43 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52 > GiB 5.2 TiB 69.97 1.28 160 up osd.43 > 44 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 > GiB 5.3 TiB 69.35 1.27 133 up osd.44 > 45 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 12 GiB 55 > GiB 4.8 TiB 72.12 1.32 148 up osd.45 > 46 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52 > GiB 5.1 TiB 70.74 1.29 148 up osd.46 > 47 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 188 MiB 54 > GiB 4.9 TiB 71.74 1.31 151 up osd.47 > -11 208.09680 - 208 TiB 147 TiB 135 TiB 27 GiB 627 > GiB 62 TiB 70.45 1.29 - host hbgt-ceph1-osd05 > 48 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 49 > GiB 5.7 TiB 67.11 1.23 152 up osd.48 > 49 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 52 > GiB 5.1 TiB 70.87 1.30 155 up osd.49 > 50 hdd 17.34140 0.90002 17 TiB 12 TiB 11 TiB 11 GiB 54 > GiB 5.2 TiB 70.01 1.28 147 up osd.50 > 51 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 84 MiB 54 > GiB 4.6 TiB 73.24 1.34 147 up osd.51 > 52 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 337 MiB 53 > GiB 4.8 TiB 72.26 1.32 146 up osd.52 > 53 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.6 GiB 54 > GiB 5.1 TiB 70.56 1.29 132 up osd.53 > 54 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.3 GiB 56 > GiB 4.5 TiB 73.77 1.35 138 up osd.54 > 55 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 10 GiB 53 > GiB 4.8 TiB 72.59 1.33 148 up osd.55 > 56 hdd 17.34140 1.00000 17 TiB 11 TiB 9.6 TiB 1.8 GiB 46 > GiB 6.7 TiB 61.16 1.12 127 up osd.56 > 57 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 3.1 MiB 50 > GiB 5.6 TiB 67.96 1.24 148 up osd.57 > 58 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 687 KiB 52 > GiB 5.0 TiB 70.93 1.30 152 up osd.58 > 59 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 22 KiB 55 > GiB 4.4 TiB 74.88 1.37 147 up osd.59 > -13 245.93866 - 246 TiB 112 TiB 106 TiB 28 GiB 498 > GiB 134 TiB 45.56 0.83 - host hbgt-ceph1-osd06 > 60 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 10 GiB 51 > GiB 9.3 TiB 54.43 0.99 134 up osd.60 > 61 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 216 MiB 43 > GiB 11 TiB 47.06 0.86 120 up osd.61 > 62 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 330 MiB 41 > GiB 11 TiB 44.53 0.81 116 up osd.62 > 63 hdd 20.49489 1.00000 20 TiB 8.9 TiB 8.4 TiB 354 MiB 39 > GiB 12 TiB 43.29 0.79 113 up osd.63 > 64 hdd 20.49489 1.00000 20 TiB 8.1 TiB 7.7 TiB 148 MiB 36 > GiB 12 TiB 39.72 0.73 117 up osd.64 > 65 hdd 20.49489 1.00000 20 TiB 12 TiB 12 TiB 14 KiB 53 > GiB 8.4 TiB 58.90 1.08 159 up osd.65 > 66 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 18 MiB 49 > GiB 9.8 TiB 52.33 0.96 129 up osd.66 > 67 hdd 20.49489 1.00000 20 TiB 498 GiB 1.5 GiB 1 KiB 745 > MiB 20 TiB 2.37 0.04 6 up osd.67 > 68 hdd 20.49489 1.00000 20 TiB 10 TiB 9.8 TiB 13 GiB 47 > GiB 10 TiB 50.12 0.92 138 up osd.68 > 69 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 2.0 GiB 48 > GiB 9.7 TiB 52.77 0.96 135 up osd.69 > 70 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 1.6 GiB 50 > GiB 9.2 TiB 55.24 1.01 150 up osd.70 > 71 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 86 MiB 42 > GiB 11 TiB 45.96 0.84 130 up osd.71 > -15 245.93866 - 246 TiB 114 TiB 108 TiB 62 GiB 505 > GiB 132 TiB 46.18 0.84 - host hbgt-ceph1-osd07 > 72 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 12 GiB 49 > GiB 9.6 TiB 53.39 0.98 146 up osd.72 > 73 hdd 20.49489 1.00000 20 TiB 8.5 TiB 8.0 TiB 194 MiB 38 > GiB 12 TiB 41.31 0.75 112 up osd.73 > 74 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 10 GiB 42 > GiB 11 TiB 46.12 0.84 124 up osd.74 > 75 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 12 GiB 41 > GiB 11 TiB 44.16 0.81 125 up osd.75 > 76 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 114 MiB 41 > GiB 11 TiB 45.91 0.84 111 up osd.76 > 77 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 1.5 GiB 42 > GiB 11 TiB 47.03 0.86 127 up osd.77 > 78 hdd 20.49489 1.00000 20 TiB 7.7 TiB 7.3 TiB 12 GiB 36 > GiB 13 TiB 37.77 0.69 105 up osd.78 > 79 hdd 20.49489 1.00000 20 TiB 8.2 TiB 7.7 TiB 1 KiB 36 > GiB 12 TiB 39.92 0.73 91 up osd.79 > 80 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 371 MiB 41 > GiB 11 TiB 46.13 0.84 132 up osd.80 > 81 hdd 20.49489 1.00000 20 TiB 9.8 TiB 9.3 TiB 353 MiB 45 > GiB 11 TiB 47.95 0.88 121 up osd.81 > 82 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 12 GiB 51 > GiB 9.2 TiB 55.23 1.01 142 up osd.82 > 83 hdd 20.49489 1.00000 20 TiB 10 TiB 9.6 TiB 1.9 GiB 44 > GiB 10 TiB 49.28 0.90 125 up osd.83 > -17 251.76105 - 252 TiB 23 TiB 11 TiB 43 MiB 64 > GiB 229 TiB 8.99 0.16 - host hbgt-ceph1-osd08 > 84 hdd 20.98009 1.00000 21 TiB 1.0 TiB 65 GiB 1 KiB 1.4 > GiB 20 TiB 4.93 0.09 5 up osd.84 > 85 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 813 > MiB 20 TiB 4.63 0.08 7 up osd.85 > 86 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.0 > GiB 20 TiB 4.63 0.08 2 up osd.86 > 87 hdd 20.98009 1.00000 21 TiB 1008 GiB 14 GiB 1 KiB 1.0 > GiB 20 TiB 4.69 0.09 7 up osd.87 > 88 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.6 > GiB 20 TiB 4.63 0.08 6 up osd.88 > 89 hdd 20.98009 1.00000 21 TiB 4.9 TiB 4.0 TiB 17 MiB 20 > GiB 16 TiB 23.56 0.43 51 up osd.89 > 90 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.5 > GiB 20 TiB 4.63 0.08 5 up osd.90 > 91 hdd 20.98009 1.00000 21 TiB 3.3 TiB 2.3 TiB 1 KiB 11 > GiB 18 TiB 15.70 0.29 40 up osd.91 > 92 hdd 20.98009 1.00000 21 TiB 5.1 TiB 4.1 TiB 26 MiB 20 > GiB 16 TiB 24.34 0.44 60 up osd.92 > 93 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.3 > GiB 20 TiB 4.63 0.08 9 up osd.93 > 94 hdd 20.98009 1.00000 21 TiB 1.4 TiB 489 GiB 1 KiB 3.4 > GiB 20 TiB 6.90 0.13 14 up osd.94 > 95 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1016 > MiB 20 TiB 4.63 0.08 4 up osd.95 > TOTAL 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1 > TiB 808 TiB 54.72 > > > I tried: > - update backfill and recovery settings > - change osd mclock profile > - change pg/pgp_num > > The cluster does not recovery. Clients are working well. > > What can I do ? > > Thanks for your help > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |