Hi, Sorry to ping this old thread, but we have a few kernel client nodes stuck like this after an outage on their network. MDS's are running v14.2.11 and the client has kernel 3.10.0-1127.19.1.el7.x86_64. This is the first time at our lab that clients didn't reconnect after a network issue (but this might be the first large client network outage after we upgraded from luminous to nautilus). It looks identical to Florian's issue: Feb 08 10:07:23 hpc-qcd027.cern.ch kernel: libceph: mds0 10.32.5.17:6821 socket closed (con state NEGOTIATING) Feb 08 10:07:51 hpc-qcd027.cern.ch kernel: ceph: get_quota_realm: ino (10004fe5035.fffffffffffffffe) null i_snap_realm The full kernel log | grep ceph is at https://termbin.com/zdwc As of now, this client's mountpoint is "stuck" and it does not have a session open on mds.0, but has sessions on mds.1 and mds.2 (see below [1]). I evicted this client from all mds's but the client didn't manage to reconnect: Feb 08 10:20:01 hpc-qcd027.cern.ch kernel: libceph: mds1 188.185.88.47:6801 socket closed (con state OPEN) Feb 08 10:20:01 hpc-qcd027.cern.ch kernel: libceph: mds2 188.185.88.90:6801 socket closed (con state OPEN) Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: mds1 188.185.88.47:6801 connection reset Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: reset on mds1 Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds1 closed our session Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds1 reconnect start Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: mds2 188.185.88.90:6801 connection reset Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: libceph: reset on mds2 Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds2 closed our session Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds2 reconnect start Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds1 reconnect denied Feb 08 10:20:02 hpc-qcd027.cern.ch kernel: ceph: mds2 reconnect denied Feb 08 10:20:21 hpc-qcd027.cern.ch kernel: ceph: get_quota_realm: ino (10004fe5035.fffffffffffffffe) null i_snap_realm Feb 08 10:20:51 hpc-qcd027.cern.ch kernel: ceph: get_quota_realm: ino (10004fe5035.fffffffffffffffe) null i_snap_realm Here are some logs from mds.0: # egrep '10.32.3.150|137564444' /var/log/ceph/ceph-mds.cephflax-mds-ca21a8a1c6.log 2021-02-08 09:16:46.875 7f9b22faa700 0 log_channel(cluster) log [WRN] : evicting unresponsive client hpc-qcd027.cern.ch:hpc (137564444), after 304.536 seconds 2021-02-08 09:17:28.326 7f9b28a6c700 0 --1- [v2:188.184.96.191:6800/1781566860,v1:188.184.96.191:6801/1781566860] >> v1:10.32.3.150:0/3998218413 conn(0x562ac988b800 0x562e1a6e8800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept we reset (peer sent cseq 1), sending RESETSESSION 2021-02-08 09:17:28.628 7f9b28a6c700 0 --1- [v2:188.184.96.191:6800/1781566860,v1:188.184.96.191:6801/1781566860] >> v1:10.32.3.150:0/3998218413 conn(0x5629ce1c6800 0x56274776f000 :6801 s=OPENED pgs=26571 cs=1 l=0).fault server, going to standby 2021-02-08 09:49:56.318 7f9b28a6c700 0 --1- [v2:188.184.96.191:6800/1781566860,v1:188.184.96.191:6801/1781566860] >> v1:10.32.3.150:0/3998218413 conn(0x5629f5eaa800 0x5627460ab800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept peer reset, then tried to connect to us, replacing compared with mds.1 where the reconnect succeeded: # egrep '10.32.3.150|137564444' /var/log/ceph/ceph-mds.cephflax-mds-370212ad58.log 2021-02-08 09:16:42.970 7fc98299b700 0 log_channel(cluster) log [WRN] : evicting unresponsive client hpc-qcd027.cern.ch:hpc (137564444), after 300.629 seconds 2021-02-08 09:17:28.327 7fc987c2f700 0 --1- [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >> v1:10.32.3.150:0/3998218413 conn(0x55a652fea400 0x55a7220ba800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept we reset (peer sent cseq 1), sending RESETSESSION 2021-02-08 09:17:28.414 7fc987c2f700 0 --1- [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >> v1:10.32.3.150:0/3998218413 conn(0x55a791481000 0x55a6c7a2d800 :6801 s=OPENED pgs=26573 cs=1 l=0).fault server, going to standby 2021-02-08 10:05:12.810 7fc988430700 0 --1- [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >> v1:10.32.3.150:0/3998218413 conn(0x55a75789d400 0x55a78695b000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2021-02-08 10:05:13.374 7fc97e192700 2 mds.1.server New client session: addr="v1:10.32.3.150:0/3998218413",elapsed=0.057278,throttled=0.000007,status="ACCEPTED",root="/hpcqcd" 2021-02-08 10:20:01.666 7fc98499f700 1 mds.1.242924 Evicting client session 137564444 (v1 10.32.3.150:0/3998218413) 2021-02-08 10:20:01.666 7fc98499f700 0 log_channel(cluster) log [INF] : Evicting client session 137564444 (v1:10.32.3.150:0/3998218413) 2021-02-08 10:20:02.343 7fc988430700 0 --1- [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >> v1:10.32.3.150:0/3998218413 conn(0x55a7a5d81c00 0x55a7823d1000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept we reset (peer sent cseq 2), sending RESETSESSION 2021-02-08 10:20:02.345 7fc988430700 0 --1- [v2:188.185.88.47:6800/3666946863,v1:188.185.88.47:6801/3666946863] >> v1:10.32.3.150:0/3998218413 conn(0x55a75bb6a000 0x55a744f69800 :6801 s=OPENED pgs=26588 cs=1 l=0).fault server, going to standby We have this in the mds config: mds session blacklist on evict = false mds session blacklist on timeout = false The clients are working fine after they are rebooted. Given the age of this thread -- maybe is this a known issue and already solved in newer kernels? Note that during this incident a few clients also crashed and rebooted -- we are still trying to get the kernel backtrace for those cases, to see if it matches https://tracker.ceph.com/issues/40862. Thanks! Dan [1] session ls: mds.cephflax-mds-ca21a8a1c6: [] mds.cephflax-mds-370212ad58: [ { "id": 137564444, "entity": { "name": { "type": "client", "num": 137564444 }, "addr": { "type": "v1", "addr": "10.32.3.150:0", "nonce": 3998218413 } }, "state": "open", "num_leases": 0, "num_caps": 0, "request_load_avg": 0, "uptime": 844.92158032299994, "requests_in_flight": 0, "completed_requests": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 0, "halflife": 300 }, "inst": "client.137564444 v1:10.32.3.150:0/3998218413", "completed_requests": [], "prealloc_inos": [], "used_inos": [], "client_metadata": { "features": "0x00000000000000ff", "entity_id": "hpc", "hostname": "hpc-qcd027.cern.ch", "kernel_version": "3.10.0-1127.19.1.el7.x86_64", "root": "/hpcqcd" } } ] mds.cephflax-mds-adccf51169: [ { "id": 137564444, "entity": { "name": { "type": "client", "num": 137564444 }, "addr": { "type": "v1", "addr": "10.32.3.150:0", "nonce": 3998218413 } }, "state": "open", "num_leases": 0, "num_caps": 0, "request_load_avg": 0, "uptime": 1761.964491447, "requests_in_flight": 0, "completed_requests": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 0, "halflife": 300 }, "inst": "client.137564444 v1:10.32.3.150:0/3998218413", "completed_requests": [], "prealloc_inos": [], "used_inos": [], "client_metadata": { "features": "0x00000000000000ff", "entity_id": "hpc", "hostname": "hpc-qcd027.cern.ch", "kernel_version": "3.10.0-1127.19.1.el7.x86_64", "root": "/hpcqcd" } } ] On Mon, Oct 14, 2019 at 12:03 PM Florian Pritz <florian.pritz@xxxxxxxxxxxxxx> wrote: > > On Wed, Oct 02, 2019 at 10:24:41PM +0800, "Yan, Zheng" <ukernel@xxxxxxxxx> wrote: > > Can you reproduce this. If you can, run 'ceph daemon mds.x session ls' > > before restart mds. > > I just managed to run into this issue again. 'ceph daemon mds.x session > ls' doesn't work because apparently our setup doesn't have the admin > socket in the expected place. I've therefore used 'ceph tell mds.0 > session ls' which I think should be the same expect for how the daemon > is contacted. > > When the issue happens and 2 clients are hanging, 'ceph tell mds.0 > session ls' shows only 9 clients instead of 11. The hanging clients are > missing from the list. Once they are rebooted they show up in the > output. > > On a potentially interesting note: The clients that were hanging this > time are the same ones as last time. They aren't set up any differently > from the others as far as I can tell though. > > Florian > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |