Hello all, The ceph-mds servers in our cluster are performing a constant boot->replay->crash in our systems. I have enable debug logging for the mds for a restart cycle on one of the nodes[1]. Kernel debug from cephfs client during reconnection attempts: [732586.352173] ceph: mdsc delayed_work [732586.352178] ceph: check_delayed_caps [732586.352182] ceph: lookup_mds_session ffff88202f01c000 210 [732586.352185] ceph: mdsc get_session ffff88202f01c000 210 -> 211 [732586.352189] ceph: send_renew_caps ignoring mds0 (up:replay) [732586.352192] ceph: add_cap_releases ffff88202f01c000 mds0 extra 680 [732586.352195] ceph: mdsc put_session ffff88202f01c000 211 -> 210 [732586.352198] ceph: mdsc delayed_work [732586.352200] ceph: check_delayed_caps [732586.352202] ceph: lookup_mds_session ffff881036cbf800 1 [732586.352205] ceph: mdsc get_session ffff881036cbf800 1 -> 2 [732586.352207] ceph: send_renew_caps ignoring mds0 (up:replay) [732586.352210] ceph: add_cap_releases ffff881036cbf800 mds0 extra 680 [732586.352212] ceph: mdsc put_session ffff881036cbf800 2 -> 1 [732591.357123] ceph: mdsc delayed_work [732591.357128] ceph: check_delayed_caps [732591.357132] ceph: lookup_mds_session ffff88202f01c000 210 [732591.357135] ceph: mdsc get_session ffff88202f01c000 210 -> 211 [732591.357139] ceph: add_cap_releases ffff88202f01c000 mds0 extra 680 [732591.357142] ceph: mdsc put_session ffff88202f01c000 211 -> 210 [732591.357145] ceph: mdsc delayed_work [732591.357147] ceph: check_delayed_caps [732591.357149] ceph: lookup_mds_session ffff881036cbf800 1 [732591.357152] ceph: mdsc get_session ffff881036cbf800 1 -> 2 [732591.357154] ceph: add_cap_releases ffff881036cbf800 mds0 extra 680 [732591.357157] ceph: mdsc put_session ffff881036cbf800 2 -> 1 [732596.362076] ceph: mdsc delayed_work [732596.362081] ceph: check_delayed_caps [732596.362084] ceph: lookup_mds_session ffff88202f01c000 210 [732596.362087] ceph: mdsc get_session ffff88202f01c000 210 -> 211 [732596.362091] ceph: add_cap_releases ffff88202f01c000 mds0 extra 680 [732596.362094] ceph: mdsc put_session ffff88202f01c000 211 -> 210 [732596.362097] ceph: mdsc delayed_work [732596.362099] ceph: check_delayed_caps [732596.362101] ceph: lookup_mds_session ffff881036cbf800 1 [732596.362104] ceph: mdsc get_session ffff881036cbf800 1 -> 2 [732596.362106] ceph: add_cap_releases ffff881036cbf800 mds0 extra 680 [732596.362109] ceph: mdsc put_session ffff881036cbf800 2 -> 1 Anybody have any debugging tips, or have any ideas on how to get an mds stable? Server info: CentOS 7.1 with Ceph 0.94.1 Client info: Gentoo, kernel cephfs. 3.19.5-gentoo I'd reboot the client, but at this point, I don't believe this is a client issue. [1] https://drive.google.com/file/d/0B4XF1RWjuGh5WU1OZXpNb0Z1ck0/view?usp=sharing -- Adam _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |