pgs stuck in 'incomplete' state, blocked ops, query command hangs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi cephers,

We have two pgs that are stuck in 'incomplete' state across two different pools: 
pg 2.525 is stuck inactive since forever, current state incomplete, last acting [55,89]
pg 0.527 is stuck inactive since forever, current state incomplete, last acting [55,89]
pg 0.527 is stuck unclean since forever, current state incomplete, last acting [55,89]
pg 2.525 is stuck unclean since forever, current state incomplete, last acting [55,89]
pg 0.527 is incomplete, acting [55,89]
pg 2.525 is incomplete, acting [55,89]

Basically, we ran into a problem where we had 2x replication and 2 disks on different machines died near-simultaneously, and my pgs were stuck in 'down+peering'. I had to do some combination of declaring the OSDs as lost, and running 'force_create_pg'. I realize the data on those pgs is now lost, but I'm stuck as to how to get the pgs out of 'incomplete'. 

I also see many ops blocked on the primary OSD for these:
100 ops are blocked > 67108.9 sec
100 ops are blocked > 67108.9 sec on osd.55

However, this is a new disk. If I 'ceph osd out osd.55', the pgs move to another OSD and the new primary gets blocked ops. Restarting osd.55 does nothing. Other pgs on osd.55 seem okay.

I would attach the result of a query, but If I run a 'ceph pg 2.525 query', the command totally hangs until I ctrl-c

ceph pg 2.525 query
^CError EINTR: problem getting command descriptions from pg.2.525

I've also tried 'ceph pg repair 2.525', which does nothing.

Any thoughts here? Are my pools totally sunk? 

Thanks,
Lincoln
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux