Hi Adam, You can get the MDS to spit out more debug information like so: # ceph mds tell 0 injectargs '--debug-mds 20 --debug-ms 1' At least then you can see where it's at when it crashes. --Lincoln On May 22, 2015, at 9:33 AM, Adam Tygart wrote: > Hello all, > > The ceph-mds servers in our cluster are performing a constant > boot->replay->crash in our systems. > > I have enable debug logging for the mds for a restart cycle on one of > the nodes[1]. > > Kernel debug from cephfs client during reconnection attempts: > [732586.352173] ceph: mdsc delayed_work > [732586.352178] ceph: check_delayed_caps > [732586.352182] ceph: lookup_mds_session ffff88202f01c000 210 > [732586.352185] ceph: mdsc get_session ffff88202f01c000 210 -> 211 > [732586.352189] ceph: send_renew_caps ignoring mds0 (up:replay) > [732586.352192] ceph: add_cap_releases ffff88202f01c000 mds0 extra 680 > [732586.352195] ceph: mdsc put_session ffff88202f01c000 211 -> 210 > [732586.352198] ceph: mdsc delayed_work > [732586.352200] ceph: check_delayed_caps > [732586.352202] ceph: lookup_mds_session ffff881036cbf800 1 > [732586.352205] ceph: mdsc get_session ffff881036cbf800 1 -> 2 > [732586.352207] ceph: send_renew_caps ignoring mds0 (up:replay) > [732586.352210] ceph: add_cap_releases ffff881036cbf800 mds0 extra 680 > [732586.352212] ceph: mdsc put_session ffff881036cbf800 2 -> 1 > [732591.357123] ceph: mdsc delayed_work > [732591.357128] ceph: check_delayed_caps > [732591.357132] ceph: lookup_mds_session ffff88202f01c000 210 > [732591.357135] ceph: mdsc get_session ffff88202f01c000 210 -> 211 > [732591.357139] ceph: add_cap_releases ffff88202f01c000 mds0 extra 680 > [732591.357142] ceph: mdsc put_session ffff88202f01c000 211 -> 210 > [732591.357145] ceph: mdsc delayed_work > [732591.357147] ceph: check_delayed_caps > [732591.357149] ceph: lookup_mds_session ffff881036cbf800 1 > [732591.357152] ceph: mdsc get_session ffff881036cbf800 1 -> 2 > [732591.357154] ceph: add_cap_releases ffff881036cbf800 mds0 extra 680 > [732591.357157] ceph: mdsc put_session ffff881036cbf800 2 -> 1 > [732596.362076] ceph: mdsc delayed_work > [732596.362081] ceph: check_delayed_caps > [732596.362084] ceph: lookup_mds_session ffff88202f01c000 210 > [732596.362087] ceph: mdsc get_session ffff88202f01c000 210 -> 211 > [732596.362091] ceph: add_cap_releases ffff88202f01c000 mds0 extra 680 > [732596.362094] ceph: mdsc put_session ffff88202f01c000 211 -> 210 > [732596.362097] ceph: mdsc delayed_work > [732596.362099] ceph: check_delayed_caps > [732596.362101] ceph: lookup_mds_session ffff881036cbf800 1 > [732596.362104] ceph: mdsc get_session ffff881036cbf800 1 -> 2 > [732596.362106] ceph: add_cap_releases ffff881036cbf800 mds0 extra 680 > [732596.362109] ceph: mdsc put_session ffff881036cbf800 2 -> 1 > > Anybody have any debugging tips, or have any ideas on how to get an mds stable? > > Server info: CentOS 7.1 with Ceph 0.94.1 > Client info: Gentoo, kernel cephfs. 3.19.5-gentoo > > I'd reboot the client, but at this point, I don't believe this is a > client issue. > > [1] https://drive.google.com/file/d/0B4XF1RWjuGh5WU1OZXpNb0Z1ck0/view?usp=sharing > > -- > Adam > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |