Ceph Luminous 12.2.5 with filestore OSDs
I have a cluster that had a bunch of disks removed due to failures and
hardware problems. At this point, after about a few day of
rebalancing and attempting to get healthy, it still has 16 incomplete
pgs that I cannot seem to get fixed.
I've tried moving some of the pgs to other osds using the
ceph-objecstore-tool. I've restarted some of the osds. Ive tried all
of the tricks I could find online for clearing these issues but they
persist.
One problem appears to be that a lot of the osds are stuck or blocked
waiting for osds that no longer exist in the crush map. 'ceph osd
blocked-by' shows many osds that are not in the cluster anymore. Is
there anyway to force the osds that are stuck waiting for non-existent
osds to move on and drop them from their list ? Even restarting them
does not fix the issue. Is it a bug that osds are blocking on
non-existent osds?
OBJECT_MISPLACED 610200/41085645 objects misplaced (1.485%)
PG_AVAILABILITY Reduced data availability: 16 pgs inactive, 3 pgs
peering, 13 pgs incomplete
pg 1.10e is stuck peering since forever, current state peering,
last acting [52,23,20]
pg 1.12a is incomplete, acting [27,63,53] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.20b is incomplete, acting [84,59,18] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.24f is incomplete, acting [13,23,19] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.25c is incomplete, acting [23,52,60] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.2bd is incomplete, acting [59,53,19] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.2e4 is incomplete, acting [67,22,6] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.2fd is stuck peering since forever, current state peering,
last acting [79,53,58]
pg 1.390 is incomplete, acting [81,18,2] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.482 is incomplete, acting [1,53,90] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.504 is incomplete, acting [59,96,53] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.688 is incomplete, acting [36,53,49] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.6dd is incomplete, acting [47,56,12] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.703 is incomplete, acting [47,2,51] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
pg 1.7a2 is stuck peering since forever, current state peering,
last acting [18,82,3]
pg 1.7b4 is incomplete, acting [92,49,96] (reducing pool
cephfs_data min_size from 2 may help; search ceph.com/docs for
'incomplete')
PG_DEGRADED Degraded data redundancy: 13439/41085645 objects degraded
(0.033%), 2 pgs degraded, 2 pgs undersized
pg 1.74 is stuck undersized for 620.126459, current state
active+undersized+degraded+remapped+backfill_wait, last acting [17,6]
pg 1.527 is stuck undersized for 712.173611, current state
active+undersized+degraded+remapped+backfill_wait, last acting [63,86]
REQUEST_SLOW 2 slow requests are blocked > 32 sec
2 ops are blocked > 2097.15 sec
osd.18 has blocked requests > 2097.15 sec
REQUEST_STUCK 63 stuck requests are blocked > 4096 sec
22 ops are blocked > 134218 sec
2 ops are blocked > 67108.9 sec
28 ops are blocked > 8388.61 sec
11 ops are blocked > 4194.3 sec
osds 23,92 have stuck requests > 4194.3 sec
osds 59,81 have stuck requests > 8388.61 sec
osd.13 has stuck requests > 67108.9 sec
osds 1,36,47,67,84 have stuck requests > 134218 sec
Any help would be much appreciated.
Wyllys Ingersoll
Keeper Technology, LLC
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
