Hi,
We had a disk on the cluster that was not responding properly and
causing 'slow requests'. The osd on the disk was stopped and the
osd was marked down and then out. Rebalancing succeeded but
(some?) pgs from that osd are now stuck in stale+active+clean
state, which is not being resolved (see below for query results).
My question: is it better to mark this osd as "lost" (i.e. 'ceph
osd lost 14') or to remove the osd as detailed here:
https://www.sebastien-han.fr/blog/2015/12/11/ceph-properly-remove-an-osd/
Thanks,
David
$ ceph health detail
HEALTH_ERR 17 pgs are stuck inactive for more than 300 seconds;
17 pgs stale; 17 pgs stuck stale
pg 7.f3 is stuck stale for 6138.330316, current state
stale+active+clean, last acting [14]
pg 7.bd is stuck stale for 6138.330365, current state
stale+active+clean, last acting [14]
pg 7.b6 is stuck stale for 6138.330374, current state
stale+active+clean, last acting [14]
pg 7.c5 is stuck stale for 6138.330363, current state
stale+active+clean, last acting [14]
pg 7.ac is stuck stale for 6138.330385, current state
stale+active+clean, last acting [14]
pg 7.5b is stuck stale for 6138.330678, current state
stale+active+clean, last acting [14]
pg 7.1b4 is stuck stale for 6138.330409, current state
stale+active+clean, last acting [14]
pg 7.182 is stuck stale for 6138.330445, current state
stale+active+clean, last acting [14]
pg 7.1f8 is stuck stale for 6138.330720, current state
stale+active+clean, last acting [14]
pg 7.53 is stuck stale for 6138.330697, current state
stale+active+clean, last acting [14]
pg 7.1d2 is stuck stale for 6138.330663, current state
stale+active+clean, last acting [14]
pg 7.70 is stuck stale for 6138.330742, current state
stale+active+clean, last acting [14]
pg 7.14f is stuck stale for 6138.330585, current state
stale+active+clean, last acting [14]
pg 7.23 is stuck stale for 6138.330610, current state
stale+active+clean, last acting [14]
pg 7.153 is stuck stale for 6138.330600, current state
stale+active+clean, last acting [14]
pg 7.cc is stuck stale for 6138.330409, current state
stale+active+clean, last acting [14]
pg 7.16b is stuck stale for 6138.330509, current state
stale+active+clean, last acting [14]
$ ceph pg dump_stuck stale
ok
pg_stat state up up_primary acting
acting_primary
7.f3 stale+active+clean [14] 14 [14] 14
7.bd stale+active+clean [14] 14 [14] 14
7.b6 stale+active+clean [14] 14 [14] 14
7.c5 stale+active+clean [14] 14 [14] 14
7.ac stale+active+clean [14] 14 [14] 14
7.5b stale+active+clean [14] 14 [14] 14
7.1b4 stale+active+clean [14] 14 [14] 14
7.182 stale+active+clean [14] 14 [14] 14
7.1f8 stale+active+clean [14] 14 [14] 14
7.53 stale+active+clean [14] 14 [14] 14
7.1d2 stale+active+clean [14] 14 [14] 14
7.70 stale+active+clean [14] 14 [14] 14
7.14f stale+active+clean [14] 14 [14] 14
7.23 stale+active+clean [14] 14 [14] 14
7.153 stale+active+clean [14] 14 [14] 14
7.cc stale+active+clean [14] 14 [14] 14
7.16b stale+active+clean [14] 14 [14] 14
