Hi, after one PG finish backfilling another PG will start to backfill. You can rise osd_max_backfills if you want to backfill more at the same time. backfill_toofull will decrease in time. Why you see toofull ? Ceph remove old data only then new are in place. While it haven`t done he calculate at full space requirement to move the data. ----- Original Message ----- From: Deep Dish <deeepdish@xxxxxxxxx> To: ceph-users@xxxxxxx Date: Wednesday, February 26, 2025, 2:47:20 PM Subject: backfill_toofull not clearing on Reef > Hello, > I have an 80 OSD cluster (across 8 nodes). The average utilization across > my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was > replaced (same capacity). For the past week or so the cluster has been > recovering, slowly, and reporting backfill_toofull. I can't figure out > what's causing the issue given there's ample available capacity. > I tried setting the following without any result: > ceph osd set-backfillfull-ratio 0.90 > # ceph -s > ... > health: HEALTH_WARN > Low space hindering backfill (add storage if this doesn't > resolve itself): 137 pgs backfill_toofull > 11 pgs not deep-scrubbed in time > ... > data: > volumes: 1/1 healthy > pools: 13 pools, 1665 pgs > objects: 27.81M objects, 79 TiB > usage: 197 TiB used, 413 TiB / 610 TiB avail > pgs: 16341449/128092605 objects misplaced (12.758%) > 1371 active+clean > 150 active+remapped+backfill_wait > 121 active+remapped+backfill_wait+backfill_toofull > 16 active+remapped+backfill_toofull > 6 active+clean+scrubbing+deep > 1 active+remapped+backfilling > io: > client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr > recovery: 16 MiB/s, 4 objects/s > # ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS > 1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9 > TiB 24.28 0.75 108 up > 9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6 > TiB 36.76 1.14 103 up > 16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1 > TiB 29.82 0.92 109 up > 27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7 > TiB 26.23 0.81 108 up > 35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6 > TiB 35.51 1.10 113 up > 43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5 > TiB 37.92 1.17 106 up > 51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8 > TiB 33.63 1.04 108 up > 59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9 > TiB 32.70 1.01 101 up > 67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9 > TiB 31.99 0.99 112 up > 74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2 > TiB 28.68 0.89 110 up > 3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5 > TiB 28.45 0.88 112 up > 11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1 > TiB 21.50 0.67 115 up > 19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6 > TiB 36.51 1.13 115 up > 25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2 > TiB 29.08 0.90 91 up > 33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2 > TiB 27.27 0.84 109 up > 41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5 > TiB 24.36 0.75 105 up > 49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1 > TiB 29.98 0.93 107 up > 56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0 > TiB 31.77 0.98 116 up > 62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5 > TiB 38.47 1.19 108 up > 69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1 > TiB 30.05 0.93 109 up > 5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2 > TiB 31.40 0.97 109 up > 13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3 > TiB 31.12 0.96 116 up > 22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8 > TiB 34.72 1.07 114 up > 31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0 > TiB 31.08 0.96 101 up > 39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7 > TiB 35.60 1.10 110 up > 47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0 > TiB 31.89 0.99 108 up > 55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4 > TiB 39.43 1.22 116 up > 65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9 > TiB 32.62 1.01 116 up > 73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6 > TiB 36.13 1.12 111 up > 78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7 > TiB 35.50 1.10 118 up > 4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6 > TiB 27.11 0.84 131 up > 12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9 > TiB 23.80 0.74 106 up > 20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4 > TiB 26.11 0.81 106 up > 28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5 > TiB 38.59 1.19 121 up > 36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9 > TiB 31.98 0.99 108 up > 44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8 > TiB 33.68 1.04 90 up > 52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1 > TiB 29.76 0.92 108 up > 60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0 > TiB 31.04 0.96 104 up > 68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8 > TiB 33.83 1.05 119 up > 76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7 > TiB 35.77 1.11 107 up > 6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1 > TiB 33.01 1.02 111 up > 14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3 > TiB 30.96 0.96 84 up > 23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8 > TiB 33.96 1.05 95 up > 30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5 > TiB 38.67 1.20 93 up > 38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0 > TiB 31.72 0.98 101 up > 46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7 > TiB 36.05 1.12 81 up > 53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6 > TiB 36.12 1.12 88 up > 63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2 > TiB 42.94 1.33 87 up > 75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6 > TiB 63.47 1.96 356 up > 79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6 > TiB 36.90 1.14 93 up > 7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1 > TiB 22.22 0.69 98 up > 15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5 > TiB 28.50 0.88 114 up > 21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6 > TiB 22.60 0.70 88 up > 29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8 > TiB 34.20 1.06 114 up > 37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9 > TiB 33.02 1.02 128 up > 45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6 > TiB 36.26 1.12 109 up > 54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7 > TiB 35.04 1.08 111 up > 61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2 > TiB 27.87 0.86 103 up > 70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0 > TiB 31.86 0.99 109 up > 77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8 > TiB 32.35 1.00 112 up > 2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2 > TiB 32.27 1.00 115 up > 10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9 > TiB 24.28 0.75 97 up > 18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3 > TiB 26.63 0.82 107 up > 26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6 > TiB 35.60 1.10 111 up > 34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6 > TiB 36.17 1.12 106 up > 42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1 > TiB 30.17 0.93 111 up > 50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6 > TiB 36.05 1.12 101 up > 58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4 > TiB 39.11 1.21 122 up > 66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3 > TiB 26.73 0.83 98 up > 72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0 > TiB 29.57 0.92 110 up > 0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3 > TiB 30.63 0.95 119 up > 8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7 > TiB 26.77 0.83 112 up > 17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5 > TiB 38.79 1.20 107 up > 24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9 > TiB 32.22 1.00 116 up > 32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7 > TiB 35.56 1.10 94 up > 40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3 > TiB 39.44 1.22 106 up > 48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9 > TiB 32.22 1.00 114 up > 57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2 > TiB 28.40 0.88 95 up > 64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6 > TiB 37.42 1.16 130 up > 71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9 > TiB 32.01 0.99 105 up > TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413 > TiB 32.31 > MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72 > # ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 86.15359 root default > -19 10.00000 host 01stor28 > 1 hdd 1.00000 osd.1 up 1.00000 1.00000 > 9 hdd 1.00000 osd.9 up 1.00000 1.00000 > 16 hdd 1.00000 osd.16 up 1.00000 1.00000 > 27 hdd 1.00000 osd.27 up 1.00000 1.00000 > 35 hdd 1.00000 osd.35 up 1.00000 1.00000 > 43 hdd 1.00000 osd.43 up 1.00000 1.00000 > 51 hdd 1.00000 osd.51 up 1.00000 1.00000 > 59 hdd 1.00000 osd.59 up 1.00000 1.00000 > 67 hdd 1.00000 osd.67 up 1.00000 1.00000 > 74 hdd 1.00000 osd.74 up 1.00000 1.00000 > -21 10.00000 host 01stor32 > 3 hdd 1.00000 osd.3 up 1.00000 1.00000 > 11 hdd 1.00000 osd.11 up 1.00000 1.00000 > 19 hdd 1.00000 osd.19 up 1.00000 1.00000 > 25 hdd 1.00000 osd.25 up 1.00000 1.00000 > 33 hdd 1.00000 osd.33 up 1.00000 1.00000 > 41 hdd 1.00000 osd.41 up 1.00000 1.00000 > 49 hdd 1.00000 osd.49 up 1.00000 1.00000 > 56 hdd 1.00000 osd.56 up 1.00000 1.00000 > 62 hdd 1.00000 osd.62 up 1.00000 1.00000 > 69 hdd 1.00000 osd.69 up 1.00000 1.00000 > -15 10.00000 host 02stor23 > 5 hdd 1.00000 osd.5 up 1.00000 1.00000 > 13 hdd 1.00000 osd.13 up 1.00000 1.00000 > 22 hdd 1.00000 osd.22 up 1.00000 1.00000 > 31 hdd 1.00000 osd.31 up 1.00000 1.00000 > 39 hdd 1.00000 osd.39 up 1.00000 1.00000 > 47 hdd 1.00000 osd.47 up 1.00000 1.00000 > 55 hdd 1.00000 osd.55 up 1.00000 1.00000 > 65 hdd 1.00000 osd.65 up 1.00000 1.00000 > 73 hdd 1.00000 osd.73 up 1.00000 1.00000 > 78 hdd 1.00000 osd.78 up 1.00000 1.00000 > -11 10.00000 host 02stor27 > 4 hdd 1.00000 osd.4 up 1.00000 1.00000 > 12 hdd 1.00000 osd.12 up 1.00000 1.00000 > 20 hdd 1.00000 osd.20 up 1.00000 1.00000 > 28 hdd 1.00000 osd.28 up 1.00000 1.00000 > 36 hdd 1.00000 osd.36 up 1.00000 1.00000 > 44 hdd 1.00000 osd.44 up 1.00000 1.00000 > 52 hdd 1.00000 osd.52 up 1.00000 1.00000 > 60 hdd 1.00000 osd.60 up 1.00000 1.00000 > 68 hdd 1.00000 osd.68 up 1.00000 1.00000 > 76 hdd 1.00000 osd.76 up 1.00000 1.00000 > -17 16.15359 host 03stor23 > 6 hdd 1.00000 osd.6 up 1.00000 1.00000 > 14 hdd 1.00000 osd.14 up 1.00000 1.00000 > 23 hdd 1.00000 osd.23 up 1.00000 1.00000 > 30 hdd 1.00000 osd.30 up 1.00000 1.00000 > 38 hdd 1.00000 osd.38 up 1.00000 1.00000 > 46 hdd 1.00000 osd.46 up 1.00000 1.00000 > 53 hdd 1.00000 osd.53 up 1.00000 1.00000 > 63 hdd 1.00000 osd.63 up 1.00000 1.00000 > 75 hdd 7.15359 osd.75 up 1.00000 1.00000 > 79 hdd 1.00000 osd.79 up 1.00000 1.00000 > -13 10.00000 host 03stor27 > 7 hdd 1.00000 osd.7 up 1.00000 1.00000 > 15 hdd 1.00000 osd.15 up 1.00000 1.00000 > 21 hdd 1.00000 osd.21 up 1.00000 1.00000 > 29 hdd 1.00000 osd.29 up 1.00000 1.00000 > 37 hdd 1.00000 osd.37 up 1.00000 1.00000 > 45 hdd 1.00000 osd.45 up 1.00000 1.00000 > 54 hdd 1.00000 osd.54 up 1.00000 1.00000 > 61 hdd 1.00000 osd.61 up 1.00000 1.00000 > 70 hdd 1.00000 osd.70 up 1.00000 1.00000 > 77 hdd 1.00000 osd.77 up 1.00000 1.00000 > -3 10.00000 host 04stor28 > 2 hdd 1.00000 osd.2 up 1.00000 1.00000 > 10 hdd 1.00000 osd.10 up 1.00000 1.00000 > 18 hdd 1.00000 osd.18 up 1.00000 1.00000 > 26 hdd 1.00000 osd.26 up 1.00000 1.00000 > 34 hdd 1.00000 osd.34 up 1.00000 1.00000 > 42 hdd 1.00000 osd.42 up 1.00000 1.00000 > 50 hdd 1.00000 osd.50 up 1.00000 1.00000 > 58 hdd 1.00000 osd.58 up 1.00000 1.00000 > 66 hdd 1.00000 osd.66 up 1.00000 1.00000 > 72 hdd 1.00000 osd.72 up 1.00000 1.00000 > -7 10.00000 host 04stor32 > 0 hdd 1.00000 osd.0 up 1.00000 1.00000 > 8 hdd 1.00000 osd.8 up 1.00000 1.00000 > 17 hdd 1.00000 osd.17 up 1.00000 1.00000 > 24 hdd 1.00000 osd.24 up 1.00000 1.00000 > 32 hdd 1.00000 osd.32 up 1.00000 1.00000 > 40 hdd 1.00000 osd.40 up 1.00000 1.00000 > 48 hdd 1.00000 osd.48 up 1.00000 1.00000 > 57 hdd 1.00000 osd.57 up 1.00000 1.00000 > 64 hdd 1.00000 osd.64 up 1.00000 1.00000 > 71 hdd 1.00000 osd.71 up 1.00000 1.00000 > # ceph health detail > HEALTH_WARN Low space hindering backfill (add storage if this doesn't > resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this > doesn't resolve itself): 137 pgs backfill_toofull > pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting > [0,67,4] > pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting > [71,44,22] > pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting > [58,51,23] > pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting > [69,45,22] > pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting > [24,3,63] > pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting > [35,50,14] > pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting > [28,16,69] > pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6] > pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting > [78,26,29] > pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30] > pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting > [43,11,17] > pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting > [68,33,2] > pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66] > pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting > [78,17,6] > pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting > [57,67,13] > pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38] > pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting > [1,0,28] > pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting > [47,14,66] > pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14] > pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting > [27,45,38] > pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72] > pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting > [52,69,46] > pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting > [55,26,32] > pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting > [34,59,11] > pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2] > pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting > [55,70,46] > pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23] > pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting > [37,25,59] > pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23] > pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting > [11,26,24] > pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting > [59,60,46] > pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6] > pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting > [35,57,46] > pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting > [28,9,72] > pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting > [5,43,72] > pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting > [70,72,14] > pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting > [78,10,23] > pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting > [66,7,53] > pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13] > pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting > [25,15,63] > pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting > [78,76,34] > pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting > [24,76,54,16,25,79,22,2] > pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting > [10,9,56,4,47,61,32,79] > pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting > [70,41,36,48,23,31,1,2] > pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting > [10,29,78,25,1,44,64,79] > pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting > [28,26,15,23,57,16,25,78] > pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting > [59,46,19,7,72,68,55,8] > pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting > [76,54,25,31,0,46,58,59] > pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting > [31,76,1,66,8,56,79,21] > pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting > [74,0,70,44,39,34,33,6] > pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting > [44,21,43,18,69,64,38,31] > [WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time > pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000 > pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000 > pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000 > pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000 > pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000 > pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000 > pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000 > pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000 > pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000 > pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000 > pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000 > pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000 > pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000 > pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000 > pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000 > pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000 > pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000 > pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000 > pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000 > pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000 > pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000 > pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000 > pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000 > pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000 > pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |