A bit more digging, the original crash appears to be similar (but not exactly the same) as this tracker report http://tracker.ceph.com/issues/16983 I can see that this was fixed in 10.2.3, so I will probably look to upgrade. If the logs make sense to anybody with a bit more knowledge I would be interested if that bug is related or if I have stumbled on something new. Nick > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Nick Fisk > Sent: 02 November 2016 17:58 > To: 'Ceph Users' <ceph-users@xxxxxxxxxxxxxx> > Subject: MDS Problems - Solved but reporting for benefit of others > > Hi all, > > Just a bit of an outage with CephFS around the MDS's, I managed to get everything up and running again after a bit of head scratching > and thought I would share here what happened. > > Cause > I believe the MDS's which were running as VM's suffered when the hypervisor ran out of ram and started swapping due to hypervisor > maintenance. I know this is less than ideal and have put steps in place to prevent this happening again. > > Symptoms > 1. Noticed that both MDS's were down, log files on both showed that they had crashed 2. After restarting MDS's, their status kept > flipping between replay and reconnect 3. Now again both MDS's would crash again 4. Log files showed they seemed to keep restarting > after trying to reconnect clients 5. Clients were all kernel one was 3.19 and the rest 4.8. I believe the problematic client was one of the > ones running Kernel 4.8 6. Ceph is 10.2.2 > > Resolution > After some serious head scratching and a little bit of panicking, the fact the log files showed the restart always happened after trying > to reconnect the clients gave me the idea to try and kill the sessions on the MDS. I first reset all the clients and waited, but this didn't > seem to have any effect and I could still see the MDS trying to reconnect to the clients. I then decided to try and kill the sessions from > the MDS end, so I shutdown the standby MDS (as they kept flipping active roles) and ran > > ceph daemon mds.gp-ceph-mds1 session ls > > I then tried to kill the last session in the list > > ceph daemon mds.gp-ceph-mds1 session evict <session id> > > I had to keep hammering this command to get it at the right point, as the MDS was only responding for a fraction of a second. > > Suddenly in my other window, where I had the tail of the MDS log, I saw a whizz of new information and then stopping with the MDS > success message. So it seems something the MDS was trying to do whilst reconnecting was upsetting it. Ceph -s updated so show > MDS was now active. Rebooting other MDS then corrected made it standby as well. Problem solved. > > I have uploaded the 2 MDS logs here if any CephFS dev's are interested in taking a closer look. > > http://app.sys-pro.co.uk/mds_logs.zip > > Nick > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |