Perhaps as a workaround you should just wipe this mon's data dir and remake it? In the past when I upgraded our mons from spinning disks to SSDs, I went through a procedure to remake each mon from scratch (wiping and resyncing each mon's leveldb one at a time). I did something like this: service ceph stop cd /var/lib/ceph/mon mv mon.4 mon.4.old mkdir mon.4 ceph-mon --mkfs -i 4 --keyring /var/lib/ceph/tmp/keyring.mon.4 (where the keyring in the last step is a copy of the mon keyring). When you start the mon after this it should get the latest leveldb from the other mons. Maybe a procedure like this will work for you?? (Disclaimer: I've only ever tried this on Dumpling). Cheers, Dan ________________________________________ From: ceph-users-bounces@xxxxxxxxxxxxxx [ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of Jens Kristian Søgaard [jens@xxxxxxxxxxxxxxxxxxxx] Sent: 31 March 2014 20:55 To: Gregory Farnum Cc: ceph-users Subject: Re: Mon hangs when started after Emperor upgrade Hi Gregory, > Is the mon process doing anything (that is, does it have any CPU > usage)? This looks to be an internal leveldb issue, but not one that > we've run into before, so I think there must be something unique about > the leveldb store involved. No, it is not doing anything at all. I'm not sure what could be unique about my leveldb store. The only thing I can think of is that this cluster was initially installed with Argonaut, and has over time been upgraded through Bobtail, Cuttlefish, Dumpling and now Emperor. -- Jens Kristian Søgaard, Mermaid Consulting ApS, jens@xxxxxxxxxxxxxxxxxxxx, http://www.mermaidconsulting.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |