Yeah, get a ceph pg query on one of the stuck ones. -Sam On Tue, 2015-03-10 at 14:41 +0000, joel.merrick@xxxxxxxxx wrote: > Stuck unclean and stuck inactive. I can fire up a full query and > health dump somewhere useful if you want (full pg query info on ones > listed in health detail, tree, osd dump etc). There were blocked_by > operations that no longer exist after doing the OSD addition. > > Side note, spent some time yesterday writing some bash to do this > programatically (might be useful to others, will throw on github) > > On Tue, Mar 10, 2015 at 1:41 PM, Samuel Just <sjust@xxxxxxxxxx> wrote: > > What do you mean by "unblocked" but still "stuck"? > > -Sam > > > > On Mon, 2015-03-09 at 22:54 +0000, joel.merrick@xxxxxxxxx wrote: > >> On Mon, Mar 9, 2015 at 2:28 PM, Samuel Just <sjust@xxxxxxxxxx> wrote: > >> > You'll probably have to recreate osds with the same ids (empty ones), > >> > let them boot, stop them, and mark them lost. There is a feature in the > >> > tracker to improve this behavior: http://tracker.ceph.com/issues/10976 > >> > -Sam > >> > >> Thanks Sam, I've readded the OSDs, they became unblocked but there are > >> still the same number of pgs stuck. I looked at them in some more > >> detail and it seems they all have num_bytes='0'. Tried a repair too, > >> for good measure. Still nothing I'm afraid. > >> > >> Does this mean some underlying catastrophe has happened and they are > >> never going to recover? Following on, would that cause data loss. > >> There are no missing objects and I'm hoping there's appropriate > >> checksumming / replicas to balance that out, but now I'm not so sure. > >> > >> Thanks again, > >> Joel > > > > > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |