Hmm, last time we saw this it meant that the MDS log had gotten corrupted somehow and was a little short (in that case due to the OSDs filling up). What do you mean by "rebuilt the OSDs"? -Greg On Mon, Nov 17, 2014 at 12:52 PM, JIten Shah <jshah2005@xxxxxx> wrote: > After i rebuilt the OSD’s, the MDS went into the degraded mode and will not > recover. > > > [jshah@Lab-cephmon001 ~]$ sudo tail -100f > /var/log/ceph/ceph-mds.Lab-cephmon001.log > 2014-11-17 17:55:27.855861 7fffef5d3700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/838757053 pipe(0x1e18000 sd=22 :6800 s=0 pgs=0 cs=0 l=0 > c=0x1e02c00).accept peer addr is really X.X.16.114:0/838757053 (socket is > X.X.16.114:34672/0) > 2014-11-17 17:57:27.855519 7fffef5d3700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/838757053 pipe(0x1e18000 sd=22 :6800 s=2 pgs=2 cs=1 l=0 > c=0x1e02c00).fault with nothing to send, going to standby > 2014-11-17 17:58:47.883799 7fffef3d1700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/26738200 pipe(0x1e1be80 sd=23 :6800 s=0 pgs=0 cs=0 l=0 > c=0x1e04ba0).accept peer addr is really X.X.16.114:0/26738200 (socket is > X.X.16.114:34699/0) > 2014-11-17 18:00:47.882484 7fffef3d1700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/26738200 pipe(0x1e1be80 sd=23 :6800 s=2 pgs=2 cs=1 l=0 > c=0x1e04ba0).fault with nothing to send, going to standby > 2014-11-17 18:01:47.886662 7fffef1cf700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/3673954317 pipe(0x1e1c380 sd=24 :6800 s=0 pgs=0 cs=0 l=0 > c=0x1e05540).accept peer addr is really X.X.16.114:0/3673954317 (socket is > X.X.16.114:34718/0) > 2014-11-17 18:03:47.885488 7fffef1cf700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/3673954317 pipe(0x1e1c380 sd=24 :6800 s=2 pgs=2 cs=1 l=0 > c=0x1e05540).fault with nothing to send, going to standby > 2014-11-17 18:04:47.888983 7fffeefcd700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/3403131574 pipe(0x1e18a00 sd=25 :6800 s=0 pgs=0 cs=0 l=0 > c=0x1e05280).accept peer addr is really X.X.16.114:0/3403131574 (socket is > X.X.16.114:34744/0) > 2014-11-17 18:06:47.888427 7fffeefcd700 0 -- X.X.16.111:6800/3046050 >> > X.X.16.114:0/3403131574 pipe(0x1e18a00 sd=25 :6800 s=2 pgs=2 cs=1 l=0 > c=0x1e05280).fault with nothing to send, going to standby > 2014-11-17 20:02:03.558250 7ffff07de700 -1 mds.0.1 *** got signal Terminated > *** > 2014-11-17 20:02:03.558297 7ffff07de700 1 mds.0.1 suicide. wanted > down:dne, now up:active > 2014-11-17 20:02:56.053339 7ffff7fe77a0 0 ceph version 0.80.5 > (38b73c67d375a2552d8ed67843c8a65c2c0feba6), process ceph-mds, pid 3424727 > 2014-11-17 20:02:56.121367 7ffff30e4700 1 mds.-1.0 handle_mds_map standby > 2014-11-17 20:02:56.124343 7ffff30e4700 1 mds.0.2 handle_mds_map i am now > mds.0.2 > 2014-11-17 20:02:56.124345 7ffff30e4700 1 mds.0.2 handle_mds_map state > change up:standby --> up:replay > 2014-11-17 20:02:56.124348 7ffff30e4700 1 mds.0.2 replay_start > 2014-11-17 20:02:56.124359 7ffff30e4700 1 mds.0.2 recovery set is > 2014-11-17 20:02:56.124362 7ffff30e4700 1 mds.0.2 need osdmap epoch 93, > have 92 > 2014-11-17 20:02:56.124363 7ffff30e4700 1 mds.0.2 waiting for osdmap 93 > (which blacklists prior instance) > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |