Hey Joffrey, try to switch back to the wpq scheduler in ceph.conf: osd_op_queue = wpq ...and restart all OSDs. I also had issues where recovery was very very slow (10kb/s). Best Regards, Alex Walender Am 17.10.24 um 11:44 schrieb Joffrey:
HI,
This is my cluster:
cluster:
id: c300532c-51fa-11ec-9a41-0050569c3b55
health: HEALTH_WARN
Degraded data redundancy: 2062374/1331064781 objects degraded
(0.155%), 278 pgs degraded, 40 pgs undersized
2497 pgs not deep-scrubbed in time
2497 pgs not scrubbed in time
services:
mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3
(age 9d)
mgr: hbgt-ceph1-mon3.gmfzqm(active, since 10d), standbys:
hbgt-ceph1-mon2.nteihj, hbgt-ceph1-mon1.thrnnu
osd: 96 osds: 96 up (since 9d), 96 in (since 45h); 1588 remapped pgs
rgw: 3 daemons active (3 hosts, 2 zones)
data:
pools: 16 pools, 2497 pgs
objects: 266.22M objects, 518 TiB
usage: 976 TiB used, 808 TiB / 1.7 PiB avail
pgs: 2062374/1331064781 objects degraded (0.155%)
349917519/1331064781 objects misplaced (26.289%)
1312 active+remapped+backfill_wait
864 active+clean
199 active+recovery_wait+degraded+remapped
38 active+recovery_wait+degraded
33 active+undersized+degraded+remapped+backfill_wait
33 active+recovery_wait+remapped
7 active+recovery_wait
6 active+undersized+degraded+remapped+backfilling
2 active+recovering+remapped
1 active+remapped+backfilling
1 active+recovering+degraded+remapped
1 active+recovery_wait+undersized+degraded+remapped
io:
client: 683 KiB/s rd, 2.2 KiB/s wr, 51 op/s rd, 2 op/s wr
No recovery is running and I don't understand why.
I have free space:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 1784.12231 - 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1
TiB 808 TiB 54.72 1.00 - root default
-5 208.09680 - 208 TiB 142 TiB 130 TiB 51 GiB 605
GiB 66 TiB 68.14 1.25 - host hbgt-ceph1-osd01
1 hdd 17.34140 1.00000 17 TiB 11 TiB 11 TiB 33 KiB 49
GiB 5.9 TiB 66.16 1.21 136 up osd.1
3 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 23 GiB 49
GiB 6.3 TiB 63.80 1.17 139 up osd.3
5 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 139 MiB 53
GiB 4.8 TiB 72.31 1.32 142 up osd.5
7 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 51
GiB 5.6 TiB 67.97 1.24 145 up osd.7
9 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 2.2 GiB 49
GiB 6.0 TiB 65.67 1.20 140 up osd.9
11 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 329 MiB 50
GiB 5.5 TiB 68.42 1.25 145 up osd.11
13 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52
GiB 5.1 TiB 70.45 1.29 153 up osd.13
15 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 61 KiB 48
GiB 5.7 TiB 66.85 1.22 144 up osd.15
17 hdd 17.34140 1.00000 17 TiB 11 TiB 9.5 TiB 272 MiB 45
GiB 6.8 TiB 60.63 1.11 120 up osd.17
19 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 12 GiB 50
GiB 5.9 TiB 65.90 1.20 134 up osd.19
21 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.6 GiB 57
GiB 4.1 TiB 76.49 1.40 152 up osd.21
23 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 31 KiB 54
GiB 4.7 TiB 73.10 1.34 124 up osd.23
-3 208.09680 - 208 TiB 146 TiB 134 TiB 64 GiB 629
GiB 62 TiB 70.05 1.28 - host hbgt-ceph1-osd02
0 hdd 17.34140 1.00000 17 TiB 11 TiB 9.8 TiB 22 GiB 49
GiB 6.6 TiB 62.07 1.13 124 up osd.0
2 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.7 GiB 52
GiB 5.2 TiB 70.14 1.28 150 up osd.2
4 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 48
GiB 5.8 TiB 66.83 1.22 152 up osd.4
6 hdd 17.34140 0.85004 17 TiB 13 TiB 12 TiB 11 GiB 58
GiB 4.0 TiB 76.85 1.40 153 up osd.6
8 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 54
GiB 4.9 TiB 71.58 1.31 152 up osd.8
10 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 6.3 MiB 47
GiB 6.1 TiB 64.91 1.19 133 up osd.10
12 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 109 MiB 51
GiB 5.6 TiB 67.72 1.24 137 up osd.12
14 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 GiB 53
GiB 5.1 TiB 70.37 1.29 148 up osd.14
16 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 2.9 GiB 54
GiB 4.9 TiB 71.65 1.31 145 up osd.16
18 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 88 MiB 55
GiB 4.8 TiB 72.45 1.32 154 up osd.18
20 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 108 MiB 55
GiB 4.6 TiB 73.39 1.34 166 up osd.20
22 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 209 MiB 54
GiB 4.7 TiB 72.68 1.33 138 up osd.22
-7 208.09680 - 208 TiB 148 TiB 136 TiB 29 GiB 635
GiB 60 TiB 71.03 1.30 - host hbgt-ceph1-osd03
24 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.4 GiB 56
GiB 4.5 TiB 74.27 1.36 150 up osd.24
25 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 32 KiB 57
GiB 4.1 TiB 76.27 1.39 162 up osd.25
26 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 1.4 GiB 49
GiB 6.0 TiB 65.30 1.19 134 up osd.26
27 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.7 GiB 53
GiB 4.7 TiB 73.16 1.34 152 up osd.27
28 hdd 17.34140 1.00000 17 TiB 12 TiB 12 TiB 2.3 MiB 52
GiB 4.9 TiB 71.97 1.32 158 up osd.28
29 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 10 GiB 52
GiB 5.3 TiB 69.72 1.27 141 up osd.29
30 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 266 MiB 54
GiB 4.6 TiB 73.38 1.34 142 up osd.30
31 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 93 MiB 50
GiB 5.7 TiB 67.36 1.23 145 up osd.31
32 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 13 GiB 50
GiB 6.1 TiB 64.77 1.18 131 up osd.32
33 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 12 MiB 53
GiB 5.2 TiB 69.77 1.28 132 up osd.33
34 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 242 MiB 50
GiB 5.5 TiB 68.17 1.25 139 up osd.34
35 hdd 17.34140 1.00000 17 TiB 14 TiB 13 TiB 135 MiB 58
GiB 3.8 TiB 78.26 1.43 157 up osd.35
-9 208.09680 - 208 TiB 146 TiB 134 TiB 38 GiB 627
GiB 62 TiB 70.19 1.28 - host hbgt-ceph1-osd04
36 hdd 17.34140 1.00000 17 TiB 11 TiB 10 TiB 210 MiB 47
GiB 6.4 TiB 63.16 1.15 139 up osd.36
37 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 222 MiB 53
GiB 5.0 TiB 71.34 1.30 147 up osd.37
38 hdd 17.34140 0.95001 17 TiB 13 TiB 12 TiB 13 MiB 55
GiB 4.5 TiB 74.33 1.36 155 up osd.38
39 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 135 MiB 52
GiB 5.2 TiB 70.22 1.28 140 up osd.39
40 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 25 MiB 53
GiB 4.9 TiB 71.46 1.31 150 up osd.40
41 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.8 GiB 54
GiB 5.0 TiB 71.36 1.30 150 up osd.41
42 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 823 MiB 50
GiB 5.8 TiB 66.49 1.22 156 up osd.42
43 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.5 GiB 52
GiB 5.2 TiB 69.97 1.28 160 up osd.43
44 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52
GiB 5.3 TiB 69.35 1.27 133 up osd.44
45 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 12 GiB 55
GiB 4.8 TiB 72.12 1.32 148 up osd.45
46 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 11 GiB 52
GiB 5.1 TiB 70.74 1.29 148 up osd.46
47 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 188 MiB 54
GiB 4.9 TiB 71.74 1.31 151 up osd.47
-11 208.09680 - 208 TiB 147 TiB 135 TiB 27 GiB 627
GiB 62 TiB 70.45 1.29 - host hbgt-ceph1-osd05
48 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 49
GiB 5.7 TiB 67.11 1.23 152 up osd.48
49 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 150 MiB 52
GiB 5.1 TiB 70.87 1.30 155 up osd.49
50 hdd 17.34140 0.90002 17 TiB 12 TiB 11 TiB 11 GiB 54
GiB 5.2 TiB 70.01 1.28 147 up osd.50
51 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 84 MiB 54
GiB 4.6 TiB 73.24 1.34 147 up osd.51
52 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 337 MiB 53
GiB 4.8 TiB 72.26 1.32 146 up osd.52
53 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 1.6 GiB 54
GiB 5.1 TiB 70.56 1.29 132 up osd.53
54 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 1.3 GiB 56
GiB 4.5 TiB 73.77 1.35 138 up osd.54
55 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 10 GiB 53
GiB 4.8 TiB 72.59 1.33 148 up osd.55
56 hdd 17.34140 1.00000 17 TiB 11 TiB 9.6 TiB 1.8 GiB 46
GiB 6.7 TiB 61.16 1.12 127 up osd.56
57 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 3.1 MiB 50
GiB 5.6 TiB 67.96 1.24 148 up osd.57
58 hdd 17.34140 1.00000 17 TiB 12 TiB 11 TiB 687 KiB 52
GiB 5.0 TiB 70.93 1.30 152 up osd.58
59 hdd 17.34140 1.00000 17 TiB 13 TiB 12 TiB 22 KiB 55
GiB 4.4 TiB 74.88 1.37 147 up osd.59
-13 245.93866 - 246 TiB 112 TiB 106 TiB 28 GiB 498
GiB 134 TiB 45.56 0.83 - host hbgt-ceph1-osd06
60 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 10 GiB 51
GiB 9.3 TiB 54.43 0.99 134 up osd.60
61 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 216 MiB 43
GiB 11 TiB 47.06 0.86 120 up osd.61
62 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 330 MiB 41
GiB 11 TiB 44.53 0.81 116 up osd.62
63 hdd 20.49489 1.00000 20 TiB 8.9 TiB 8.4 TiB 354 MiB 39
GiB 12 TiB 43.29 0.79 113 up osd.63
64 hdd 20.49489 1.00000 20 TiB 8.1 TiB 7.7 TiB 148 MiB 36
GiB 12 TiB 39.72 0.73 117 up osd.64
65 hdd 20.49489 1.00000 20 TiB 12 TiB 12 TiB 14 KiB 53
GiB 8.4 TiB 58.90 1.08 159 up osd.65
66 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 18 MiB 49
GiB 9.8 TiB 52.33 0.96 129 up osd.66
67 hdd 20.49489 1.00000 20 TiB 498 GiB 1.5 GiB 1 KiB 745
MiB 20 TiB 2.37 0.04 6 up osd.67
68 hdd 20.49489 1.00000 20 TiB 10 TiB 9.8 TiB 13 GiB 47
GiB 10 TiB 50.12 0.92 138 up osd.68
69 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 2.0 GiB 48
GiB 9.7 TiB 52.77 0.96 135 up osd.69
70 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 1.6 GiB 50
GiB 9.2 TiB 55.24 1.01 150 up osd.70
71 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 86 MiB 42
GiB 11 TiB 45.96 0.84 130 up osd.71
-15 245.93866 - 246 TiB 114 TiB 108 TiB 62 GiB 505
GiB 132 TiB 46.18 0.84 - host hbgt-ceph1-osd07
72 hdd 20.49489 1.00000 20 TiB 11 TiB 10 TiB 12 GiB 49
GiB 9.6 TiB 53.39 0.98 146 up osd.72
73 hdd 20.49489 1.00000 20 TiB 8.5 TiB 8.0 TiB 194 MiB 38
GiB 12 TiB 41.31 0.75 112 up osd.73
74 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 10 GiB 42
GiB 11 TiB 46.12 0.84 124 up osd.74
75 hdd 20.49489 1.00000 20 TiB 9.1 TiB 8.6 TiB 12 GiB 41
GiB 11 TiB 44.16 0.81 125 up osd.75
76 hdd 20.49489 1.00000 20 TiB 9.4 TiB 8.9 TiB 114 MiB 41
GiB 11 TiB 45.91 0.84 111 up osd.76
77 hdd 20.49489 1.00000 20 TiB 9.6 TiB 9.2 TiB 1.5 GiB 42
GiB 11 TiB 47.03 0.86 127 up osd.77
78 hdd 20.49489 1.00000 20 TiB 7.7 TiB 7.3 TiB 12 GiB 36
GiB 13 TiB 37.77 0.69 105 up osd.78
79 hdd 20.49489 1.00000 20 TiB 8.2 TiB 7.7 TiB 1 KiB 36
GiB 12 TiB 39.92 0.73 91 up osd.79
80 hdd 20.49489 1.00000 20 TiB 9.5 TiB 9.0 TiB 371 MiB 41
GiB 11 TiB 46.13 0.84 132 up osd.80
81 hdd 20.49489 1.00000 20 TiB 9.8 TiB 9.3 TiB 353 MiB 45
GiB 11 TiB 47.95 0.88 121 up osd.81
82 hdd 20.49489 1.00000 20 TiB 11 TiB 11 TiB 12 GiB 51
GiB 9.2 TiB 55.23 1.01 142 up osd.82
83 hdd 20.49489 1.00000 20 TiB 10 TiB 9.6 TiB 1.9 GiB 44
GiB 10 TiB 49.28 0.90 125 up osd.83
-17 251.76105 - 252 TiB 23 TiB 11 TiB 43 MiB 64
GiB 229 TiB 8.99 0.16 - host hbgt-ceph1-osd08
84 hdd 20.98009 1.00000 21 TiB 1.0 TiB 65 GiB 1 KiB 1.4
GiB 20 TiB 4.93 0.09 5 up osd.84
85 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 813
MiB 20 TiB 4.63 0.08 7 up osd.85
86 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.0
GiB 20 TiB 4.63 0.08 2 up osd.86
87 hdd 20.98009 1.00000 21 TiB 1008 GiB 14 GiB 1 KiB 1.0
GiB 20 TiB 4.69 0.09 7 up osd.87
88 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.6
GiB 20 TiB 4.63 0.08 6 up osd.88
89 hdd 20.98009 1.00000 21 TiB 4.9 TiB 4.0 TiB 17 MiB 20
GiB 16 TiB 23.56 0.43 51 up osd.89
90 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.5
GiB 20 TiB 4.63 0.08 5 up osd.90
91 hdd 20.98009 1.00000 21 TiB 3.3 TiB 2.3 TiB 1 KiB 11
GiB 18 TiB 15.70 0.29 40 up osd.91
92 hdd 20.98009 1.00000 21 TiB 5.1 TiB 4.1 TiB 26 MiB 20
GiB 16 TiB 24.34 0.44 60 up osd.92
93 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1.3
GiB 20 TiB 4.63 0.08 9 up osd.93
94 hdd 20.98009 1.00000 21 TiB 1.4 TiB 489 GiB 1 KiB 3.4
GiB 20 TiB 6.90 0.13 14 up osd.94
95 hdd 20.98009 1.00000 21 TiB 995 GiB 1.5 GiB 1 KiB 1016
MiB 20 TiB 4.63 0.08 4 up osd.95
TOTAL 1.7 PiB 976 TiB 895 TiB 298 GiB 4.1
TiB 808 TiB 54.72
I tried:
- update backfill and recovery settings
- change osd mclock profile
- change pg/pgp_num
The cluster does not recovery. Clients are working well.
What can I do ?
Thanks for your help
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
M.Sc Alex Walender
Forschungszentrum Jülich
Institut für Bio- und Geowissenschaften
IBG 5 - Computergestützte Metagenomik /
de.NBI Cloud Site Bielefeld
Büro : Universität Bielefeld (UHG), M3-118
Tel. : +49-521-106-2907
Attachment:
OpenPGP_0xB8E94FB3F9EAFED3.asc
Description: OpenPGP public key
Attachment:
OpenPGP_signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
