Hello,
I have an 80 OSD cluster (across 8 nodes). The average utilization across
my OSDs is ~ 32%. Recently the cluster had a bad drive, and it was
replaced (same capacity). For the past week or so the cluster has been
recovering, slowly, and reporting backfill_toofull. I can't figure out
what's causing the issue given there's ample available capacity.
I tried setting the following without any result:
ceph osd set-backfillfull-ratio 0.90
# ceph -s
...
health: HEALTH_WARN
Low space hindering backfill (add storage if this doesn't
resolve itself): 137 pgs backfill_toofull
11 pgs not deep-scrubbed in time
...
data:
volumes: 1/1 healthy
pools: 13 pools, 1665 pgs
objects: 27.81M objects, 79 TiB
usage: 197 TiB used, 413 TiB / 610 TiB avail
pgs: 16341449/128092605 objects misplaced (12.758%)
1371 active+clean
150 active+remapped+backfill_wait
121 active+remapped+backfill_wait+backfill_toofull
16 active+remapped+backfill_toofull
6 active+clean+scrubbing+deep
1 active+remapped+backfilling
io:
client: 5.4 KiB/s rd, 5 op/s rd, 0 op/s wr
recovery: 16 MiB/s, 4 objects/s
# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS
1 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 720 KiB 5.8 GiB 6.9
TiB 24.28 0.75 108 up
9 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 20 MiB 8.8 GiB 4.6
TiB 36.76 1.14 103 up
16 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 63 KiB 6.1 GiB 5.1
TiB 29.82 0.92 109 up
27 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 1.9 MiB 6.5 GiB 6.7
TiB 26.23 0.81 108 up
35 hdd 1.00000 1.00000 7.2 TiB 2.5 TiB 2.5 TiB 21 MiB 7.2 GiB 4.6
TiB 35.51 1.10 113 up
43 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 38 MiB 9.8 GiB 4.5
TiB 37.92 1.17 106 up
51 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 36 MiB 7.5 GiB 4.8
TiB 33.63 1.04 108 up
59 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 20 MiB 9.0 GiB 4.9
TiB 32.70 1.01 101 up
67 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 447 KiB 6.7 GiB 4.9
TiB 31.99 0.99 112 up
74 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 1.2 MiB 6.3 GiB 5.2
TiB 28.68 0.89 110 up
3 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 18 MiB 8.0 GiB 6.5
TiB 28.45 0.88 112 up
11 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 1.9 TiB 55 MiB 6.7 GiB 7.1
TiB 21.50 0.67 115 up
19 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 16 MiB 7.7 GiB 4.6
TiB 36.51 1.13 115 up
25 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 912 KiB 6.0 GiB 5.2
TiB 29.08 0.90 91 up
33 hdd 1.00000 1.00000 7.2 TiB 2.0 TiB 2.0 TiB 22 MiB 7.1 GiB 5.2
TiB 27.27 0.84 109 up
41 hdd 1.00000 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.9 MiB 5.7 GiB 5.5
TiB 24.36 0.75 105 up
49 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 19 MiB 9.8 GiB 5.1
TiB 29.98 0.93 107 up
56 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 87 MiB 6.9 GiB 5.0
TiB 31.77 0.98 116 up
62 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 52 MiB 9.9 GiB 4.5
TiB 38.47 1.19 108 up
69 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 3.2 MiB 7.3 GiB 5.1
TiB 30.05 0.93 109 up
5 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.8 TiB 950 KiB 8.5 GiB 6.2
TiB 31.40 0.97 109 up
13 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 36 MiB 9.4 GiB 6.3
TiB 31.12 0.96 116 up
22 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 52 MiB 9.1 GiB 4.8
TiB 34.72 1.07 114 up
31 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 39 MiB 8.2 GiB 5.0
TiB 31.08 0.96 101 up
39 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 35 MiB 8.0 GiB 4.7
TiB 35.60 1.10 110 up
47 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 58 MiB 9.3 GiB 5.0
TiB 31.89 0.99 108 up
55 hdd 1.00000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 17 MiB 9.3 GiB 4.4
TiB 39.43 1.22 116 up
65 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 21 MiB 6.9 GiB 4.9
TiB 32.62 1.01 116 up
73 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.4 MiB 9.6 GiB 4.6
TiB 36.13 1.12 111 up
78 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 2.9 MiB 7.3 GiB 4.7
TiB 35.50 1.10 118 up
4 hdd 1.00000 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 19 MiB 8.1 GiB 6.6
TiB 27.11 0.84 131 up
12 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 19 MiB 6.6 GiB 6.9
TiB 23.80 0.74 106 up
20 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 35 MiB 5.4 GiB 5.4
TiB 26.11 0.81 106 up
28 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 MiB 7.6 GiB 4.5
TiB 38.59 1.19 121 up
36 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.0 GiB 4.9
TiB 31.98 0.99 108 up
44 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 34 MiB 7.6 GiB 4.8
TiB 33.68 1.04 90 up
52 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 1.0 MiB 6.1 GiB 5.1
TiB 29.76 0.92 108 up
60 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 17 MiB 8.6 GiB 5.0
TiB 31.04 0.96 104 up
68 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 56 MiB 9.0 GiB 4.8
TiB 33.83 1.05 119 up
76 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 246 KiB 7.0 GiB 4.7
TiB 35.77 1.11 107 up
6 hdd 1.00000 1.00000 9.1 TiB 3.0 TiB 3.0 TiB 928 KiB 14 GiB 6.1
TiB 33.01 1.02 111 up
14 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 65 KiB 11 GiB 6.3
TiB 30.96 0.96 84 up
23 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 9.1 GiB 4.8
TiB 33.96 1.05 95 up
30 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 18 MiB 10 GiB 4.5
TiB 38.67 1.20 93 up
38 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 18 MiB 9.2 GiB 5.0
TiB 31.72 0.98 101 up
46 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 21 MiB 11 GiB 4.7
TiB 36.05 1.12 81 up
53 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 15 MiB 9.8 GiB 4.6
TiB 36.12 1.12 88 up
63 hdd 1.00000 1.00000 7.3 TiB 3.1 TiB 3.1 TiB 19 MiB 13 GiB 4.2
TiB 42.94 1.33 87 up
75 hdd 7.15359 1.00000 7.2 TiB 4.5 TiB 4.5 TiB 158 MiB 13 GiB 2.6
TiB 63.47 1.96 356 up
79 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 79 KiB 10 GiB 4.6
TiB 36.90 1.14 93 up
7 hdd 1.00000 1.00000 9.1 TiB 2.0 TiB 2.0 TiB 49 KiB 5.5 GiB 7.1
TiB 22.22 0.69 98 up
15 hdd 1.00000 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 19 MiB 7.3 GiB 6.5
TiB 28.50 0.88 114 up
21 hdd 1.00000 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.7 MiB 5.1 GiB 5.6
TiB 22.60 0.70 88 up
29 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 15 MiB 7.0 GiB 4.8
TiB 34.20 1.06 114 up
37 hdd 1.00000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 55 MiB 8.0 GiB 4.9
TiB 33.02 1.02 128 up
45 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 559 KiB 8.3 GiB 4.6
TiB 36.26 1.12 109 up
54 hdd 1.00000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 17 MiB 7.6 GiB 4.7
TiB 35.04 1.08 111 up
61 hdd 1.00000 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 38 MiB 7.6 GiB 5.2
TiB 27.87 0.86 103 up
70 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1.7 MiB 6.9 GiB 5.0
TiB 31.86 0.99 109 up
77 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 22 MiB 8.2 GiB 4.8
TiB 32.35 1.00 112 up
2 hdd 1.00000 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 3.1 MiB 11 GiB 6.2
TiB 32.27 1.00 115 up
10 hdd 1.00000 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 531 KiB 6.6 GiB 6.9
TiB 24.28 0.75 97 up
18 hdd 1.00000 1.00000 7.3 TiB 1.9 TiB 1.9 TiB 34 MiB 5.9 GiB 5.3
TiB 26.63 0.82 107 up
26 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 18 MiB 6.8 GiB 4.6
TiB 35.60 1.10 111 up
34 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 51 MiB 8.4 GiB 4.6
TiB 36.17 1.12 106 up
42 hdd 1.00000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 39 MiB 8.3 GiB 5.1
TiB 30.17 0.93 111 up
50 hdd 1.00000 1.00000 7.2 TiB 2.6 TiB 2.6 TiB 1.1 MiB 7.6 GiB 4.6
TiB 36.05 1.12 101 up
58 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 20 MiB 9.5 GiB 4.4
TiB 39.11 1.21 122 up
66 hdd 1.00000 1.00000 7.2 TiB 1.9 TiB 1.9 TiB 19 MiB 6.5 GiB 5.3
TiB 26.73 0.83 98 up
72 hdd 1.00000 1.00000 7.2 TiB 2.1 TiB 2.1 TiB 18 MiB 7.1 GiB 5.0
TiB 29.57 0.92 110 up
0 hdd 1.00000 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 2.6 MiB 10 GiB 6.3
TiB 30.63 0.95 119 up
8 hdd 1.00000 1.00000 9.1 TiB 2.4 TiB 2.4 TiB 34 MiB 6.9 GiB 6.7
TiB 26.77 0.83 112 up
17 hdd 1.00000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 22 MiB 8.9 GiB 4.5
TiB 38.79 1.20 107 up
24 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 73 MiB 6.6 GiB 4.9
TiB 32.22 1.00 116 up
32 hdd 1.00000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 18 MiB 9.4 GiB 4.7
TiB 35.56 1.10 94 up
40 hdd 1.00000 1.00000 7.2 TiB 2.8 TiB 2.8 TiB 36 MiB 9.2 GiB 4.3
TiB 39.44 1.22 106 up
48 hdd 1.00000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 7.0 MiB 6.5 GiB 4.9
TiB 32.22 1.00 114 up
57 hdd 1.00000 1.00000 7.3 TiB 2.1 TiB 2.1 TiB 33 MiB 8.3 GiB 5.2
TiB 28.40 0.88 95 up
64 hdd 1.00000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1.5 MiB 9.8 GiB 4.6
TiB 37.42 1.16 130 up
71 hdd 1.00000 1.00000 7.2 TiB 2.3 TiB 2.3 TiB 960 KiB 7.0 GiB 4.9
TiB 32.01 0.99 105 up
TOTAL 610 TiB 197 TiB 196 TiB 1.7 GiB 651 GiB 413
TiB 32.31
MIN/MAX VAR: 0.67/1.96 STDDEV: 5.72
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 86.15359 root default
-19 10.00000 host 01stor28
1 hdd 1.00000 osd.1 up 1.00000 1.00000
9 hdd 1.00000 osd.9 up 1.00000 1.00000
16 hdd 1.00000 osd.16 up 1.00000 1.00000
27 hdd 1.00000 osd.27 up 1.00000 1.00000
35 hdd 1.00000 osd.35 up 1.00000 1.00000
43 hdd 1.00000 osd.43 up 1.00000 1.00000
51 hdd 1.00000 osd.51 up 1.00000 1.00000
59 hdd 1.00000 osd.59 up 1.00000 1.00000
67 hdd 1.00000 osd.67 up 1.00000 1.00000
74 hdd 1.00000 osd.74 up 1.00000 1.00000
-21 10.00000 host 01stor32
3 hdd 1.00000 osd.3 up 1.00000 1.00000
11 hdd 1.00000 osd.11 up 1.00000 1.00000
19 hdd 1.00000 osd.19 up 1.00000 1.00000
25 hdd 1.00000 osd.25 up 1.00000 1.00000
33 hdd 1.00000 osd.33 up 1.00000 1.00000
41 hdd 1.00000 osd.41 up 1.00000 1.00000
49 hdd 1.00000 osd.49 up 1.00000 1.00000
56 hdd 1.00000 osd.56 up 1.00000 1.00000
62 hdd 1.00000 osd.62 up 1.00000 1.00000
69 hdd 1.00000 osd.69 up 1.00000 1.00000
-15 10.00000 host 02stor23
5 hdd 1.00000 osd.5 up 1.00000 1.00000
13 hdd 1.00000 osd.13 up 1.00000 1.00000
22 hdd 1.00000 osd.22 up 1.00000 1.00000
31 hdd 1.00000 osd.31 up 1.00000 1.00000
39 hdd 1.00000 osd.39 up 1.00000 1.00000
47 hdd 1.00000 osd.47 up 1.00000 1.00000
55 hdd 1.00000 osd.55 up 1.00000 1.00000
65 hdd 1.00000 osd.65 up 1.00000 1.00000
73 hdd 1.00000 osd.73 up 1.00000 1.00000
78 hdd 1.00000 osd.78 up 1.00000 1.00000
-11 10.00000 host 02stor27
4 hdd 1.00000 osd.4 up 1.00000 1.00000
12 hdd 1.00000 osd.12 up 1.00000 1.00000
20 hdd 1.00000 osd.20 up 1.00000 1.00000
28 hdd 1.00000 osd.28 up 1.00000 1.00000
36 hdd 1.00000 osd.36 up 1.00000 1.00000
44 hdd 1.00000 osd.44 up 1.00000 1.00000
52 hdd 1.00000 osd.52 up 1.00000 1.00000
60 hdd 1.00000 osd.60 up 1.00000 1.00000
68 hdd 1.00000 osd.68 up 1.00000 1.00000
76 hdd 1.00000 osd.76 up 1.00000 1.00000
-17 16.15359 host 03stor23
6 hdd 1.00000 osd.6 up 1.00000 1.00000
14 hdd 1.00000 osd.14 up 1.00000 1.00000
23 hdd 1.00000 osd.23 up 1.00000 1.00000
30 hdd 1.00000 osd.30 up 1.00000 1.00000
38 hdd 1.00000 osd.38 up 1.00000 1.00000
46 hdd 1.00000 osd.46 up 1.00000 1.00000
53 hdd 1.00000 osd.53 up 1.00000 1.00000
63 hdd 1.00000 osd.63 up 1.00000 1.00000
75 hdd 7.15359 osd.75 up 1.00000 1.00000
79 hdd 1.00000 osd.79 up 1.00000 1.00000
-13 10.00000 host 03stor27
7 hdd 1.00000 osd.7 up 1.00000 1.00000
15 hdd 1.00000 osd.15 up 1.00000 1.00000
21 hdd 1.00000 osd.21 up 1.00000 1.00000
29 hdd 1.00000 osd.29 up 1.00000 1.00000
37 hdd 1.00000 osd.37 up 1.00000 1.00000
45 hdd 1.00000 osd.45 up 1.00000 1.00000
54 hdd 1.00000 osd.54 up 1.00000 1.00000
61 hdd 1.00000 osd.61 up 1.00000 1.00000
70 hdd 1.00000 osd.70 up 1.00000 1.00000
77 hdd 1.00000 osd.77 up 1.00000 1.00000
-3 10.00000 host 04stor28
2 hdd 1.00000 osd.2 up 1.00000 1.00000
10 hdd 1.00000 osd.10 up 1.00000 1.00000
18 hdd 1.00000 osd.18 up 1.00000 1.00000
26 hdd 1.00000 osd.26 up 1.00000 1.00000
34 hdd 1.00000 osd.34 up 1.00000 1.00000
42 hdd 1.00000 osd.42 up 1.00000 1.00000
50 hdd 1.00000 osd.50 up 1.00000 1.00000
58 hdd 1.00000 osd.58 up 1.00000 1.00000
66 hdd 1.00000 osd.66 up 1.00000 1.00000
72 hdd 1.00000 osd.72 up 1.00000 1.00000
-7 10.00000 host 04stor32
0 hdd 1.00000 osd.0 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
17 hdd 1.00000 osd.17 up 1.00000 1.00000
24 hdd 1.00000 osd.24 up 1.00000 1.00000
32 hdd 1.00000 osd.32 up 1.00000 1.00000
40 hdd 1.00000 osd.40 up 1.00000 1.00000
48 hdd 1.00000 osd.48 up 1.00000 1.00000
57 hdd 1.00000 osd.57 up 1.00000 1.00000
64 hdd 1.00000 osd.64 up 1.00000 1.00000
71 hdd 1.00000 osd.71 up 1.00000 1.00000
# ceph health detail
HEALTH_WARN Low space hindering backfill (add storage if this doesn't
resolve itself): 137 pgs backfill_toofull; 25 pgs not deep-scrubbed in time
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this
doesn't resolve itself): 137 pgs backfill_toofull
pg 3.d6 is active+remapped+backfill_wait+backfill_toofull, acting
[0,67,4]
pg 3.dd is active+remapped+backfill_wait+backfill_toofull, acting
[71,44,22]
pg 3.df is active+remapped+backfill_wait+backfill_toofull, acting
[58,51,23]
pg 3.e3 is active+remapped+backfill_wait+backfill_toofull, acting
[69,45,22]
pg 3.e9 is active+remapped+backfill_wait+backfill_toofull, acting
[24,3,63]
pg 3.ed is active+remapped+backfill_wait+backfill_toofull, acting
[35,50,14]
pg 3.ee is active+remapped+backfill_wait+backfill_toofull, acting
[28,16,69]
pg 3.f5 is active+remapped+backfill_toofull, acting [69,31,6]
pg 3.f7 is active+remapped+backfill_wait+backfill_toofull, acting
[78,26,29]
pg 3.123 is active+remapped+backfill_toofull, acting [56,5,30]
pg 3.12b is active+remapped+backfill_wait+backfill_toofull, acting
[43,11,17]
pg 3.12d is active+remapped+backfill_wait+backfill_toofull, acting
[68,33,2]
pg 3.130 is active+remapped+backfill_toofull, acting [69,55,66]
pg 3.132 is active+remapped+backfill_wait+backfill_toofull, acting
[78,17,6]
pg 3.135 is active+remapped+backfill_wait+backfill_toofull, acting
[57,67,13]
pg 3.144 is active+remapped+backfill_toofull, acting [49,40,38]
pg 3.150 is active+remapped+backfill_wait+backfill_toofull, acting
[1,0,28]
pg 3.152 is active+remapped+backfill_wait+backfill_toofull, acting
[47,14,66]
pg 3.155 is active+remapped+backfill_toofull, acting [69,17,14]
pg 3.15a is active+remapped+backfill_wait+backfill_toofull, acting
[27,45,38]
pg 3.160 is active+remapped+backfill_toofull, acting [63,40,72]
pg 3.165 is active+remapped+backfill_wait+backfill_toofull, acting
[52,69,46]
pg 3.170 is active+remapped+backfill_wait+backfill_toofull, acting
[55,26,32]
pg 3.18c is active+remapped+backfill_wait+backfill_toofull, acting
[34,59,11]
pg 3.18d is active+remapped+backfill_toofull, acting [69,55,2]
pg 3.190 is active+remapped+backfill_wait+backfill_toofull, acting
[55,70,46]
pg 3.192 is active+remapped+backfill_toofull, acting [49,32,23]
pg 3.194 is active+remapped+backfill_wait+backfill_toofull, acting
[37,25,59]
pg 3.195 is active+remapped+backfill_toofull, acting [60,67,23]
pg 3.198 is active+remapped+backfill_wait+backfill_toofull, acting
[11,26,24]
pg 3.199 is active+remapped+backfill_wait+backfill_toofull, acting
[59,60,46]
pg 3.1a7 is active+remapped+backfill_toofull, acting [49,70,6]
pg 3.1a9 is active+remapped+backfill_wait+backfill_toofull, acting
[35,57,46]
pg 3.1ad is active+remapped+backfill_wait+backfill_toofull, acting
[28,9,72]
pg 3.1c1 is active+remapped+backfill_wait+backfill_toofull, acting
[5,43,72]
pg 3.1c3 is active+remapped+backfill_wait+backfill_toofull, acting
[70,72,14]
pg 3.1c5 is active+remapped+backfill_wait+backfill_toofull, acting
[78,10,23]
pg 3.1c9 is active+remapped+backfill_wait+backfill_toofull, acting
[66,7,53]
pg 3.1cc is active+remapped+backfill_toofull, acting [69,57,13]
pg 3.1d0 is active+remapped+backfill_wait+backfill_toofull, acting
[25,15,63]
pg 3.1d4 is active+remapped+backfill_wait+backfill_toofull, acting
[78,76,34]
pg 13.d0 is active+remapped+backfill_wait+backfill_toofull, acting
[24,76,54,16,25,79,22,2]
pg 13.dc is active+remapped+backfill_wait+backfill_toofull, acting
[10,9,56,4,47,61,32,79]
pg 13.df is active+remapped+backfill_wait+backfill_toofull, acting
[70,41,36,48,23,31,1,2]
pg 13.e6 is active+remapped+backfill_wait+backfill_toofull, acting
[10,29,78,25,1,44,64,79]
pg 13.ed is active+remapped+backfill_wait+backfill_toofull, acting
[28,26,15,23,57,16,25,78]
pg 13.f3 is active+remapped+backfill_wait+backfill_toofull, acting
[59,46,19,7,72,68,55,8]
pg 13.f4 is active+remapped+backfill_wait+backfill_toofull, acting
[76,54,25,31,0,46,58,59]
pg 13.f8 is active+remapped+backfill_wait+backfill_toofull, acting
[31,76,1,66,8,56,79,21]
pg 13.fa is active+remapped+backfill_wait+backfill_toofull, acting
[74,0,70,44,39,34,33,6]
pg 13.fb is active+remapped+backfill_wait+backfill_toofull, acting
[44,21,43,18,69,64,38,31]
[WRN] PG_NOT_DEEP_SCRUBBED: 25 pgs not deep-scrubbed in time
pg 3.1c3 not deep-scrubbed since 2025-02-13T15:17:28.388957+0000
pg 3.1ad not deep-scrubbed since 2025-02-13T16:02:08.898265+0000
pg 3.193 not deep-scrubbed since 2025-02-13T22:19:36.417333+0000
pg 13.ea not deep-scrubbed since 2025-02-14T01:30:06.577214+0000
pg 13.ec not deep-scrubbed since 2025-02-13T20:37:21.733717+0000
pg 3.df not deep-scrubbed since 2025-02-13T13:45:46.138552+0000
pg 13.d4 not deep-scrubbed since 2025-02-13T23:59:25.220544+0000
pg 3.d6 not deep-scrubbed since 2025-02-14T00:21:52.193693+0000
pg 3.39 not deep-scrubbed since 2025-02-13T18:34:34.835946+0000
pg 3.34 not deep-scrubbed since 2025-02-14T02:36:39.881359+0000
pg 13.24 not deep-scrubbed since 2025-02-13T14:50:44.835921+0000
pg 13.18 not deep-scrubbed since 2025-02-13T14:52:26.614983+0000
pg 19.16 not deep-scrubbed since 2025-02-13T18:04:00.806874+0000
pg 13.12 not deep-scrubbed since 2025-02-13T16:09:07.463626+0000
pg 3.46 not deep-scrubbed since 2025-02-14T04:15:13.558869+0000
pg 13.6e not deep-scrubbed since 2025-02-14T06:12:39.967530+0000
pg 13.6b not deep-scrubbed since 2025-02-13T12:46:15.170914+0000
pg 13.68 not deep-scrubbed since 2025-02-14T00:46:57.174151+0000
pg 3.68 not deep-scrubbed since 2025-02-13T09:12:30.533273+0000
pg 19.7b not deep-scrubbed since 2025-02-13T23:25:48.927657+0000
pg 13.8b not deep-scrubbed since 2025-02-13T19:36:49.091612+0000
pg 3.8a not deep-scrubbed since 2025-02-14T00:37:09.241173+0000
pg 13.a5 not deep-scrubbed since 2025-02-14T04:50:17.140499+0000
pg 13.bf not deep-scrubbed since 2025-02-13T05:19:59.495905+0000
pg 3.b9 not deep-scrubbed since 2025-02-13T23:56:56.222456+0000
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
