Craig, Thanks for your response. I have already marked osd.6 as lost, as you suggested. The problem is that it is still querying osd.8 which is not lost. I don't know why it is stuck there. It has been querying osd.8 for 4 days now. I also tried deleting the broken RBD image but the operation just hangs. Kevin On 5/1/14 10:11 , kevin horan wrote: > Here is how I got into this state. I have only 6 OSDs total, 3 on > one host (vashti) and 3 on another (zadok). I set the noout flag > so I could reboot zadok. Zadok was down for 2 minutes. When it > came up ceph began recovering the objects that had not been > replicated yet. Before recovery finished, osd.6, on vashti, died > (IO errors on disk, whole drive un-recoverable). Since osd.6 had > objects that had not yet had a chance to replicate to any OSD on > zadok, they were lost. I cannot recover anything further from osd.6. > > I'm pretty far out of my element here, but if osd.6 is gone, it might help to mark it lost: ceph osd lost 6 I had similiar issues when I lost some PGs. I don't think that it actually fixed my issue, but marking osds as lost did help Ceph move forward. You could also try deleting the broken RBD image, and see if that helps. -- Craig Lewis Senior Systems Engineer Office +1.714.602.1309 Email clewis-04jk9TcbgGYP2IHM84UzcNBPR1lH4CV8 at public.gmane.org <mailto:clewis-04jk9TcbgGYP2IHM84UzcNBPR1lH4CV8 at public.gmane.org> Central Desktop. Work together in ways you never thought possible. Connect with us Website <http://www.centraldesktop.com/> | Twitter <http://www.twitter.com/centraldesktop> | Facebook <http://www.facebook.com/CentralDesktop> | LinkedIn <http://www.linkedin.com/groups?gid=147417> | Blog <http://cdblog.centraldesktop.com/> _______________________________________________ ceph-users mailing list ceph-users at ... <http://gmane.org/get-address.php?address=ceph%2dusers%2didqoXFIVOFJgJs9I8MT0rw%40public.gmane.org> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140503/00bf1178/attachment.htm>