Hello,
Been working on Ceph for only a few weeks and have a small cluster in VMs. I did a ceph-ansible rolling_update to nautilus from mimic and some of my PG were stuck in 'active+undersized+remapped+backfilling' with no progress.
All OSD were up and in (see ceph tree below). The PGs only had 2 OSD in the acting set, yet 3 in the UP set. I don't understand how the acting set can have two and the upset can have 3; if anything, wouldn't the upset be a subset of acting?
Anyway, I noticed that the ansible rolling_update set the following flags 'noout' AND 'norebalance'. PG query showed backfill target as OSD 0 (which was missing from the acting set) and "waiting on backfill" was blank, as such I'm very confused.
So it wants to backfill OSD 0, it's not blocked per empty set in waiting_on_backfill, so what's holding it up? Why is it not in the acting set? (what's the clear definition of acting vs up)
ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.08817 root default
-5 0.02939 host hostosd1
0 hdd 0.00980 osd.0 up 1.00000 1.00000
4 hdd 0.00980 osd.4 up 1.00000 1.00000
7 hdd 0.00980 osd.7 up 1.00000 1.00000
-3 0.02939 host hostosd2
1 hdd 0.00980 osd.1 up 1.00000 1.00000
3 hdd 0.00980 osd.3 up 1.00000 1.00000
6 hdd 0.00980 osd.6 up 1.00000 1.00000
-7 0.02939 host hostosd3
2 hdd 0.00980 osd.2 up 1.00000 1.00000
5 hdd 0.00980 osd.5 up 1.00000 1.00000
8 hdd 0.00980 osd.8 up 1.00000 1.00000
PG Info
1.35 3 0 0 0 0 8388623 0 0 3045 3045 active+undersized+remapped+backfilling 2019-05-09 16:18:02.513033 50'107145 50:108127 [5,6,0] 5 [5,6]
PG Query
"state": "active+undersized+remapped+backfilling",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 50,
"up": [
5,
6,
0
],
"acting": [
5,
6
],
"backfill_targets": [
"0"
],
"acting_recovery_backfill": [
"0",
"5",
"6"
]
...
"waiting_on_backfill": [],
"last_backfill_started": "MAX",
"backfill_info": {
"begin": "MAX",
"end": "MAX",
"objects": []
},
"peer_backfill_info": [
"0",
{
"begin": "MAX",
"end": "MAX",
"objects": []
}
],
"backfills_in_flight": [],
"recovering": [],
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
