Hi,
This is a cluster running 17.2.7 upgraded from 16.2.6 on the 15 January
2024.
On Monday 22 January we had 4 HDD all on different server with I/O-error
because of some damage sectors, the OSD is hybrid so the DB is on SSD, 5
HDD share 1 SSD.
I set the OSD out, ceph osd out 223 269 290 318 and all hell broke
loose.
I took only minutes before the users complained about Ceph not working.
Ceph status reportet slow OPS on the OSDs that was set to out, and “ceph
tell osd.<id> dump_ops_in_flight” against the out OSDs it just hang,
after 30 minutes I stopped the dump command.
Long story short I ended up running “ceph osd set nobackfill” to slow
ops was gone and then unset it when the slow ops message disappeared.
I needed to run that all the time so the cluster didn’t come to a holt
so this oneliner loop was used
“while true; do ceph -s | grep -qE "oldest one blocked for [0-9]{2,}" &&
(date; ceph osd set nobackfill; sleep 15; ceph osd unset nobackfill);
sleep 10; done”
But now 4 days later the backfilling has stopped progressing completely
and the number of misplaced object is increasing.
Some PG has 0 misplaced object but sill have backfilling state, and been
in this state for over 24 hours now.
I have a hunch that it’s because of PG 404.6e7 is in state
“active+recovering+degraded+remapped” it’s been in this state for over
48 hours.
It’s has possible 2 missing object, but since they are not unfound I
can’t delete them with “ceph pg 404.6e7 mark_unfound_lost delete”
Could someone please help to solve this?
Down below is some output of ceph commands, I’ll also attache them.
ceph status (only removed information about no running scrub and
deep_scrub)
---
cluster:
id: b321e76e-da3a-11eb-b75c-4f948441dcd0
health: HEALTH_WARN
Degraded data redundancy: 2/6294904971 objects degraded
(0.000%), 1 pg degraded
services:
mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 11d)
mgr: ceph-mon-1.ptrsea(active, since 11d), standbys:
ceph-mon-2.mfdanx
mds: 1/1 daemons up, 1 standby
osd: 355 osds: 355 up (since 22h), 351 in (since 4d); 18 remapped
pgs
rgw: 7 daemons active (7 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 14 pools, 3945 pgs
objects: 1.14G objects, 1.1 PiB
usage: 1.8 PiB used, 1.2 PiB / 3.0 PiB avail
pgs: 2/6294904971 objects degraded (0.000%)
2980455/6294904971 objects misplaced (0.047%)
3901 active+clean
22 active+clean+scrubbing+deep
17 active+remapped+backfilling
4 active+clean+scrubbing
1 active+recovering+degraded+remapped
io:
client: 167 MiB/s rd, 13 MiB/s wr, 6.02k op/s rd, 2.35k op/s wr
ceph health detail (only removed information about no running scrub and
deep_scrub)
---
HEALTH_WARN Degraded data redundancy: 2/6294902067 objects degraded
(0.000%), 1 pg degraded
[WRN] PG_DEGRADED: Degraded data redundancy: 2/6294902067 objects
degraded (0.000%), 1 pg degraded
pg 404.6e7 is active+recovering+degraded+remapped, acting
[223,274,243,290,286,283]
ceph pg 202.6e7 list_unfound
---
{
"num_missing": 2,
"num_unfound": 0,
"objects": [],
"state": "Active",
"available_might_have_unfound": true,
"might_have_unfound": [],
"more": false
}
ceph pg 404.6e7 query | jq .recovery_state
---
[
{
"name": "Started/Primary/Active",
"enter_time": "2024-01-26T09:08:41.918637+0000",
"might_have_unfound": [
{
"osd": "243(2)",
"status": "already probed"
},
{
"osd": "274(1)",
"status": "already probed"
},
{
"osd": "275(0)",
"status": "already probed"
},
{
"osd": "283(5)",
"status": "already probed"
},
{
"osd": "286(4)",
"status": "already probed"
},
{
"osd": "290(3)",
"status": "already probed"
},
{
"osd": "335(3)",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"275(0)",
"335(3)"
],
"waiting_on_backfill": [],
"last_backfill_started":
"404:e76011a9:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.18_56463c71-286c-4399-8d5d-0c278b7c97fd:head",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2024-01-26T09:08:40.909151+0000"
}
]
ceph pg ls recovering backfilling
---
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES
OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE
SINCE VERSION REPORTED UP
ACTING
404.bc 287986 0 0 0 512046716673
0 0 10091 0 active+recovering+remapped
2h 217988'1385478 217988:10897565 [193,297,279,276,136,197]p193
[223,297,269,276,136,197]p223
404.c4 288236 0 288236 0 511669837559
0 0 10063 0 active+remapped+backfilling
24h 217988'1378228 217988:11719855 [156,186,178,345,339,177]p156
[223,186,178,345,339,177]p223
404.12a 287544 0 0 0 512246100354
0 0 10009 0 active+remapped+backfilling
24h 217988'1392371 217988:13739524 [248,178,250,145,304,272]p248
[223,178,250,145,304,272]p223
404.1c1 287739 0 286969 0 511800674008
0 0 10047 0 active+remapped+backfilling
2d 217988'1402889 217988:10975174 [332,246,183,169,280,255]p332
[318,246,183,169,280,255]p318
404.258 287737 0 277111 0 510099501390
0 0 10077 0 active+remapped+backfilling
24h 217988'1451778 217988:12780104 [308,199,134,342,188,221]p308
[318,199,134,342,188,221]p318
404.269 287990 0 0 0 512343190608
0 0 10043 0 active+remapped+backfilling
24h 217988'1358446 217988:14020217 [275,205,283,247,211,292]p275
[223,205,283,247,211,292]p223
404.34e 287624 0 277899 0 510447074297
0 0 10002 0 active+remapped+backfilling
24h 217988'1392933 217988:12636557 [322,141,338,168,251,218]p322
[318,141,338,168,251,218]p318
404.39c 287844 0 286692 0 512947685682
0 0 10017 0 active+remapped+backfilling
2d 217988'1414697 217988:11004944 [288,188,131,299,295,181]p288
[318,188,131,299,295,181]p318
404.511 287589 0 0 0 512014863711
0 0 10057 0 active+remapped+backfilling
24h 217988'1368741 217988:11544729 [166,151,327,333,186,150]p166
[223,151,327,333,186,150]p223
404.5f1 288126 0 286621 0 510850256945
0 0 10071 0 active+remapped+backfilling
24h 217988'1365831 217988:10348125 [214,332,289,184,255,160]p214
[223,332,289,184,255,160]p223
404.62a 288035 0 0 0 511318662269
0 0 10014 0 active+remapped+backfilling
3h 217988'1358010 217988:12528704 [322,260,259,319,149,152]p322
[318,260,259,319,149,152]p318
404.63d 287372 0 286559 0 508783837699
0 0 10074 0 active+remapped+backfilling
24h 217988'1402174 217988:11685744 [303,307,186,350,161,267]p303
[318,307,186,350,161,267]p318
404.6e3 288110 0 0 0 509047569016
0 0 10049 0 active+remapped+backfilling
24h 217988'1368547 217988:12202278 [166,317,233,144,337,240]p166
[223,317,233,144,337,240]p223
404.6e7 287856 2 2 0 510383394904
0 0 10047 0 active+recovering+degraded+remapped
3h 217988'1356501 217988:13157749 [275,274,243,335,286,283]p275
[223,274,243,290,286,283]p223
404.7d2 287619 0 286026 0 510708533087
0 0 10093 0 active+remapped+backfilling
3d 217988'1397393 217988:12146656 [185,139,299,222,155,149]p185
[223,139,299,222,155,149]p223
412.119 711468 0 0 0 207473602580
0 0 10099 0 active+remapped+backfilling
24h 217988'21613330 217988:87589096 [352,207,292,314,230,262]p352
[318,207,292,314,230,262]p318
412.12f 711529 0 701279 0 208498170310
0 0 10033 0 active+remapped+backfilling
24h 217988'14873593 217988:86198113 [303,305,183,215,130,244]p303
[318,305,183,215,130,244]p318
412.1fb 713044 0 3166 0 207787641403
0 0 10097 0 active+remapped+backfilling
2d 217988'14893270 217988:105346132 [156,137,228,241,262,353]p156
[223,137,228,241,262,353]p223
ceph osd tree out
---
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 3112.43481 root default
-67 192.35847 host ceph-hd-001
269 hdd 12.82390 osd.269 up 0 1.00000
-49 192.35847 host ceph-hd-003
223 hdd 12.82390 osd.223 up 0 1.00000
-73 192.35847 host ceph-hd-011
290 hdd 12.82390 osd.290 up 0 1.00000
-79 192.35847 host ceph-hd-014
318 hdd 12.82390 osd.318 up 0 1.00000
cluster:
id: b321e76e-da3a-11eb-b75c-4f948441dcd0
health: HEALTH_WARN
Degraded data redundancy: 2/6294904971 objects degraded (0.000%), 1 pg degraded
services:
mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 11d)
mgr: ceph-mon-1.ptrsea(active, since 11d), standbys: ceph-mon-2.mfdanx
mds: 1/1 daemons up, 1 standby
osd: 355 osds: 355 up (since 22h), 351 in (since 4d); 18 remapped pgs
rgw: 7 daemons active (7 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 14 pools, 3945 pgs
objects: 1.14G objects, 1.1 PiB
usage: 1.8 PiB used, 1.2 PiB / 3.0 PiB avail
pgs: 2/6294904971 objects degraded (0.000%)
2980455/6294904971 objects misplaced (0.047%)
3901 active+clean
22 active+clean+scrubbing+deep
17 active+remapped+backfilling
4 active+clean+scrubbing
1 active+recovering+degraded+remapped
io:
client: 167 MiB/s rd, 13 MiB/s wr, 6.02k op/s rd, 2.35k op/s wr
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING
404.bc 287986 0 0 0 512046716673 0 0 10091 0 active+recovering+remapped 2h 217988'1385478 217988:10897565 [193,297,279,276,136,197]p193 [223,297,269,276,136,197]p223
404.c4 288236 0 288236 0 511669837559 0 0 10063 0 active+remapped+backfilling 24h 217988'1378228 217988:11719855 [156,186,178,345,339,177]p156 [223,186,178,345,339,177]p223
404.12a 287544 0 0 0 512246100354 0 0 10009 0 active+remapped+backfilling 24h 217988'1392371 217988:13739524 [248,178,250,145,304,272]p248 [223,178,250,145,304,272]p223
404.1c1 287739 0 286969 0 511800674008 0 0 10047 0 active+remapped+backfilling 2d 217988'1402889 217988:10975174 [332,246,183,169,280,255]p332 [318,246,183,169,280,255]p318
404.258 287737 0 277111 0 510099501390 0 0 10077 0 active+remapped+backfilling 24h 217988'1451778 217988:12780104 [308,199,134,342,188,221]p308 [318,199,134,342,188,221]p318
404.269 287990 0 0 0 512343190608 0 0 10043 0 active+remapped+backfilling 24h 217988'1358446 217988:14020217 [275,205,283,247,211,292]p275 [223,205,283,247,211,292]p223
404.34e 287624 0 277899 0 510447074297 0 0 10002 0 active+remapped+backfilling 24h 217988'1392933 217988:12636557 [322,141,338,168,251,218]p322 [318,141,338,168,251,218]p318
404.39c 287844 0 286692 0 512947685682 0 0 10017 0 active+remapped+backfilling 2d 217988'1414697 217988:11004944 [288,188,131,299,295,181]p288 [318,188,131,299,295,181]p318
404.511 287589 0 0 0 512014863711 0 0 10057 0 active+remapped+backfilling 24h 217988'1368741 217988:11544729 [166,151,327,333,186,150]p166 [223,151,327,333,186,150]p223
404.5f1 288126 0 286621 0 510850256945 0 0 10071 0 active+remapped+backfilling 24h 217988'1365831 217988:10348125 [214,332,289,184,255,160]p214 [223,332,289,184,255,160]p223
404.62a 288035 0 0 0 511318662269 0 0 10014 0 active+remapped+backfilling 3h 217988'1358010 217988:12528704 [322,260,259,319,149,152]p322 [318,260,259,319,149,152]p318
404.63d 287372 0 286559 0 508783837699 0 0 10074 0 active+remapped+backfilling 24h 217988'1402174 217988:11685744 [303,307,186,350,161,267]p303 [318,307,186,350,161,267]p318
404.6e3 288110 0 0 0 509047569016 0 0 10049 0 active+remapped+backfilling 24h 217988'1368547 217988:12202278 [166,317,233,144,337,240]p166 [223,317,233,144,337,240]p223
404.6e7 287856 2 2 0 510383394904 0 0 10047 0 active+recovering+degraded+remapped 3h 217988'1356501 217988:13157749 [275,274,243,335,286,283]p275 [223,274,243,290,286,283]p223
404.7d2 287619 0 286026 0 510708533087 0 0 10093 0 active+remapped+backfilling 3d 217988'1397393 217988:12146656 [185,139,299,222,155,149]p185 [223,139,299,222,155,149]p223
412.119 711468 0 0 0 207473602580 0 0 10099 0 active+remapped+backfilling 24h 217988'21613330 217988:87589096 [352,207,292,314,230,262]p352 [318,207,292,314,230,262]p318
412.12f 711529 0 701279 0 208498170310 0 0 10033 0 active+remapped+backfilling 24h 217988'14873593 217988:86198113 [303,305,183,215,130,244]p303 [318,305,183,215,130,244]p318
412.1fb 713044 0 3166 0 207787641403 0 0 10097 0 active+remapped+backfilling 2d 217988'14893270 217988:105346132 [156,137,228,241,262,353]p156 [223,137,228,241,262,353]p223
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY
404.6e7 active+recovering+degraded+remapped [275,274,243,335,286,283] 275 [223,274,243,290,286,283] 223
404.c4 active+remapped+backfilling [156,186,178,345,339,177] 156 [223,186,178,345,339,177] 223
404.12a active+remapped+backfilling [248,178,250,145,304,272] 248 [223,178,250,145,304,272] 223
404.7d2 active+remapped+backfilling [185,139,299,222,155,149] 185 [223,139,299,222,155,149] 223
404.5f1 active+remapped+backfilling [214,332,289,184,255,160] 214 [223,332,289,184,255,160] 223
412.1fb active+remapped+backfilling [156,137,228,241,262,353] 156 [223,137,228,241,262,353] 223
404.258 active+remapped+backfilling [308,199,134,342,188,221] 308 [318,199,134,342,188,221] 318
404.511 active+remapped+backfilling [166,151,327,333,186,150] 166 [223,151,327,333,186,150] 223
404.6e3 active+remapped+backfilling [166,317,233,144,337,240] 166 [223,317,233,144,337,240] 223
404.39c active+remapped+backfilling [288,188,131,299,295,181] 288 [318,188,131,299,295,181] 318
404.62a active+remapped+backfilling [322,260,259,319,149,152] 322 [318,260,259,319,149,152] 318
404.34e active+remapped+backfilling [322,141,338,168,251,218] 322 [318,141,338,168,251,218] 318
404.269 active+remapped+backfilling [275,205,283,247,211,292] 275 [223,205,283,247,211,292] 223
412.119 active+remapped+backfilling [352,207,292,314,230,262] 352 [318,207,292,314,230,262] 318
412.12f active+remapped+backfilling [303,305,183,215,130,244] 303 [318,305,183,215,130,244] 318
404.1c1 active+remapped+backfilling [332,246,183,169,280,255] 332 [318,246,183,169,280,255] 318
404.bc active+remapped+backfilling [193,297,279,276,136,197] 193 [223,297,269,276,136,197] 223
404.63d active+remapped+backfilling [303,307,186,350,161,267] 303 [318,307,186,350,161,267] 318
[
{
"name": "Started/Primary/Active",
"enter_time": "2024-01-26T09:08:41.918637+0000",
"might_have_unfound": [
{
"osd": "243(2)",
"status": "already probed"
},
{
"osd": "274(1)",
"status": "already probed"
},
{
"osd": "275(0)",
"status": "already probed"
},
{
"osd": "283(5)",
"status": "already probed"
},
{
"osd": "286(4)",
"status": "already probed"
},
{
"osd": "290(3)",
"status": "already probed"
},
{
"osd": "335(3)",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"275(0)",
"335(3)"
],
"waiting_on_backfill": [],
"last_backfill_started": "404:e76011a9:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.18_56463c71-286c-4399-8d5d-0c278b7c97fd:head",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
}
},
{
"name": "Started",
"enter_time": "2024-01-26T09:08:40.909151+0000"
}
]
{
"num_missing": 2,
"num_unfound": 0,
"objects": [],
"state": "Active",
"available_might_have_unfound": true,
"might_have_unfound": [],
"more": false
}
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 3112.43481 root default
-67 192.35847 host ceph-hd-001
269 hdd 12.82390 osd.269 up 0 1.00000
-49 192.35847 host ceph-hd-003
223 hdd 12.82390 osd.223 up 0 1.00000
-73 192.35847 host ceph-hd-011
290 hdd 12.82390 osd.290 up 0 1.00000
-79 192.35847 host ceph-hd-014
318 hdd 12.82390 osd.318 up 0 1.00000
HEALTH_WARN Degraded data redundancy: 2/6294902067 objects degraded (0.000%), 1 pg degraded
[WRN] PG_DEGRADED: Degraded data redundancy: 2/6294902067 objects degraded (0.000%), 1 pg degraded
pg 404.6e7 is active+recovering+degraded+remapped, acting [223,274,243,290,286,283]
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
