Hi, I have some wild freeze using cephfs with the kernel driver For instance: [Tue Dec 4 10:57:48 2018] libceph: mon1 10.5.0.88:6789 session lost, hunting for new mon [Tue Dec 4 10:57:48 2018] libceph: mon2 10.5.0.89:6789 session established [Tue Dec 4 10:58:20 2018] ceph: mds0 caps stale [..] server is now frozen, filesystem accesses are stuck [Tue Dec 4 11:13:02 2018] libceph: mds0 10.5.0.88:6804 socket closed (con state OPEN) [Tue Dec 4 11:13:03 2018] libceph: mds0 10.5.0.88:6804 connection reset [Tue Dec 4 11:13:03 2018] libceph: reset on mds0 [Tue Dec 4 11:13:03 2018] ceph: mds0 closed our session [Tue Dec 4 11:13:03 2018] ceph: mds0 reconnect start [Tue Dec 4 11:13:04 2018] ceph: mds0 reconnect denied [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 000000003f1ae609 1099692263746 [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 00000000ccd58b71 1099692263749 [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 00000000da5acf8f 1099692263750 [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 000000005ddc2fcf 1099692263751 [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 00000000469a70f4 1099692263754 [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 000000005c0038f9 1099692263757 [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 00000000e7288aa2 1099692263758 [Tue Dec 4 11:13:04 2018] ceph: dropping dirty+flushing Fw state for 00000000b431209a 1099692263759 [Tue Dec 4 11:13:04 2018] libceph: mds0 10.5.0.88:6804 socket closed (con state NEGOTIATING) [Tue Dec 4 11:13:31 2018] libceph: osd12 10.5.0.89:6805 socket closed (con state OPEN) [Tue Dec 4 11:13:35 2018] libceph: osd17 10.5.0.89:6800 socket closed (con state OPEN) [Tue Dec 4 11:13:35 2018] libceph: osd9 10.5.0.88:6813 socket closed (con state OPEN) [Tue Dec 4 11:13:41 2018] libceph: osd0 10.5.0.87:6800 socket closed (con state OPEN) Kernel 4.17 is used, we got the same issue with 4.18 Ceph 13.2.1 is used >From what I understand, the kernel hang itself for some reason (perhaps it simply cannot handle some wild event) Is there a fix for that ? Secondly, it seems that the kernel reconnect itself after 15 minutes everytime Where is that tunable ? Could I lower that variables, so that hang have less impacts ? On ceph.log, I get Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST), but this is probably the consequence, not the cause Any tip ? Best regards, _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |