I could, I suppose, update the monmaps in the working monitors to remove the broken ones and then re-deploy the broken ones. The main concern I have is that if the mdsmap update isn't pending on the working ones, what else isn't in sync. Thoughts? -- Adam On Fri, Sep 30, 2016 at 11:05 AM, Adam Tygart <mozes@xxxxxxx> wrote: > Hello all, > > I've gotten myself into a bit of a bind. I was prepping to add a new > mds node to my ceph cluster. e.g. ceph-deploy mds create mormo > > Unfortunately, it started the mds server before I was ready. My > cluster was running 10.2.1, and the newly deployed mds is 10.2.3. > > This caused 3 of my 5 monitors to crash. Since I immediately realized > the mds was a newer version, I took that opportunity to upgrade my > monitors to 10.2.3. Three of the 5 monitors continue to crash. And it > looks like they are crashing when trying to apply a pending mdsmap > update. > > The log is available here: > http://people.cis.ksu.edu/~mozes/hobbit01.mon-20160930.log.gz > > I have attempted (making backups of course) to extract the monmap from > a working monitor and inserting it into a broken one. No luck, and > backup was restored. > > I need to get these monitors back up post-haste. > > If you've got any ideas, I would be greatful. > > -- > Adam _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |