On Fri, Sep 30, 2016 at 11:39 AM, Adam Tygart <mozes@xxxxxxx> wrote: > Hello all, > > Not sure if this went through before or not, as I can't check the > mailing list archives. > > I've gotten myself into a bit of a bind. I was prepping to add a new > mds node to my ceph cluster. e.g. ceph-deploy mds create mormo > > Unfortunately, it started the mds server before I was ready. My > cluster was running 10.2.1, and the newly deployed mds is 10.2.3. > > This caused 3 of my 5 monitors to crash. Since I immediately realized > the mds was a newer version, I took that opportunity to upgrade my > monitors to 10.2.3. Three of the 5 monitors continue to crash. And it > looks like they are crashing when trying to apply a pending mdsmap > update. > > The log is available here: > http://people.cis.ksu.edu/~mozes/hobbit01.mon-20160930.log.gz > > I have attempted (making backups of course) to extract the monmap from > a working monitor and inserting it into a broken one. No luck, and > backup was restored. > > Since I had 2 working monitors, I backed up the monitor stores, > updated the monmaps to remove the broken ones and tried to restart > them. I then tried to restart the "working" ones. They then failed in > the same way. I've now restored my backups of those monitors. > > I need to get these monitors back up post-haste. > > If you've got any ideas, I would be grateful. I'm not sure but it looks like it's now too late to keep the problem out of the durable storage, but if you try again make sure you turn off the MDS first. It sort of looks like you've managed to get a failed MDS with an invalid fscid (ie, a cephfs filesystem ID). ...or maybe just a terrible coding mistake. As mentioned on irc, wip-fixup-mds-standby-init should fix it. I've created a ticket as well: http://tracker.ceph.com/issues/17466 -Greg > > -- > Adam > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |