Hi, I have a cephfs kernel client (Ubuntu 18.04 4.15.0-34-generic) that's completely hung after the client was evicted by the MDS. The client logged: Jan 24 17:31:46 client kernel: [10733559.309496] libceph: FULL or reached pool quota Jan 24 17:32:26 client kernel: [10733599.232213] libceph: mon0 n.n.n.n:6789 session lost, hunting for new mon And the MDS logged: 2019-01-24 17:36:38.859 7f3ac7844700 0 log_channel(cluster) log [WRN] : evicting unresponsive client client:cephfs-client (86527773), after 300.081 seconds Looking in mdsc shows: % head /sys/kernel/debug/ceph/[id].client86527773/mdsc 20 mds0 getattr #1000003d042 21 mds0 getattr #1000003d042 22 mds0 getattr #1000003d042 23 mds0 getattr #1000003d042 24 mds0 getattr #1000003d042 25 mds0 getattr #1000003d042 26 mds0 getattr #1000003d042 27 mds0 getattr #1000003d042 28 mds0 getattr #1000003d042 29 mds0 getattr #1000003d042 But osdc hangs when I try to access it. I've tried umount -f but it hangs too. umount -l hides the problem (df no longer hangs), but any processes that were trying to access the mount are still blocked. I've also tried switching back and forth to standby MDSs in case that unblocked something. There are no current OSD blacklist entries either. It looks like rebooting is the only option, but that's somewhat of a pain to do. There's lots of people using this machine :-( Any ideas? Tim. -- Tim Bishop http://www.bishnet.net/tim/ PGP Key: 0x6C226B37FDF38D55 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |