Re: mon stucks on probing and out of quorum, after down and restart

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Redeploying would probably be the fastest way if you don't want your cluster in a degraded state for too long. You can check the logs afterwards to see what went wrong. 


Zitat von mk <mk@xxxxxx>:
I have just seen that on failed mon store.db size is 50K but on both other healthy mons 151M, 
What is the best practice? redeploy failed mon?


On 10. Sep 2021, at 13:08, Eugen Block <eblock@xxxxxx> wrote:
Yes, give it a try. If the cluster is healthy otherwise it shouldn't be a problem. 


Zitat von mk <mk@xxxxxx>:


Thx Eugen,
just stopping mon and remove/rename only store.db and start mon?
BR
Max

On 10. Sep 2021, at 12:50, Eugen Block <eblock@xxxxxx> wrote:
I don't have an explanation but removing the mon store from the failed mon has resolved similar issues in the past. Could you give that a try? 


Zitat von mk <mk@xxxxxx>:


Hi CephFolks,
I have a cluster 14.2.21-22/Ubuntu 18.04 with 3 mon’s. After going down/restart of 1 mon(amon3) it stucks on probing and its also out of quorum, we have changed nothing and it was working, regardless we have checked tcp ports / mtu's of mons are open and reachable. 
appreciate any help you can provide.

ceph health detail
HEALTH_WARN 1/3 mons down, quorum amon1,amon2
MON_DOWN 1/3 mons down, quorum amon1,amon2
mon.amon3 (rank 0) addr [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] is down (out of quorum)
all mon daemons are up and running, monmaptool printout is also equal on all mon nodes and on other healthy mon nodes log there are entries from amon3: 

Log from healthy amon1 log:
2021-09-10 10:49:16.657 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/245515484' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-09-10 11:04:16.586 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/2472330429' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-09-10 11:06:06.245 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/1444461156' entity='client.admin' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-09-10 11:19:16.581 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/964654151' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 

#########

monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 14
fsid 10a40588-d89e-4fac-be88-afcedc1a7372
last_changed 2021-06-22 13:35:56.211202
created 2018-07-10 12:37:03.124657
min_mon_release 14 (nautilus)
0: [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] mon.amon3
1: [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] mon.amon1
2: [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] mon.amon2

##########

ceph config get mon.*
WHO MASK LEVEL OPTION VALUE RO mon advanced mon_crush_min_required_version firefly * 
mon      advanced mon_warn_on_insecure_global_id_reclaim_allowed false

##########

ceph --admin-daemon /var/run/ceph/ceph-mon.amon3.asok mon_status
{
  "name": "amon3",
  "rank": 0,
  "state": "probing",
  "election_epoch": 0,
  "quorum": [],
  "features": {
      "required_con": "2449958747315912708",
      "required_mon": [
          "kraken",
          "luminous",
          "mimic",
          "osdmap-prune",
          "nautilus"
      ],
      "quorum_con": "0",
      "quorum_mon": []
  },
  "outside_quorum": [
      "amon3"
  ],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": {
      "epoch": 14,
      "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
      "modified": "2021-06-22 13:35:56.211202",
      "created": "2018-07-10 12:37:03.124657",
      "min_mon_release": 14,
      "min_mon_release_name": "nautilus",
      "features": {
          "persistent": [
              "kraken",
              "luminous",
              "mimic",
              "osdmap-prune",
              "nautilus"
          ],
          "optional": []
      },
      "mons": [
          {
              "rank": 0,
              "name": "amon3",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.83:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.83:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.83:6789/0",
              "public_addr": "172.16.12.83:6789/0"
          },
          {
              "rank": 1,
              "name": "amon1",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.81:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.81:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.81:6789/0",
              "public_addr": "172.16.12.81:6789/0"
          },
          {
              "rank": 2,
              "name": "amon2",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.82:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.82:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.82:6789/0",
              "public_addr": "172.16.12.82:6789/0"
          }
      ]
  },
  "feature_map": {
      "mon": [
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 1
          }
      ]
  }
}

#########


ceph --admin-daemon /var/run/ceph/ceph-mon.amon2.asok mon_status
{
  "name": "amon2",
  "rank": 2,
  "state": "peon",
  "election_epoch": 4612,
  "quorum": [
      1,
      2
  ],
  "quorum_age": 65650,
  "features": {
      "required_con": "2449958747315912708",
      "required_mon": [
          "kraken",
          "luminous",
          "mimic",
          "osdmap-prune",
          "nautilus"
      ],
      "quorum_con": "4611087854035861503",
      "quorum_mon": [
          "kraken",
          "luminous",
          "mimic",
          "osdmap-prune",
          "nautilus"
      ]
  },
  "outside_quorum": [],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": {
      "epoch": 14,
      "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
      "modified": "2021-06-22 11:35:56.211202",
      "created": "2018-07-10 10:37:03.124657",
      "min_mon_release": 14,
      "min_mon_release_name": "nautilus",
      "features": {
          "persistent": [
              "kraken",
              "luminous",
              "mimic",
              "osdmap-prune",
              "nautilus"
          ],
          "optional": []
      },
      "mons": [
          {
              "rank": 0,
              "name": "amon3",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.83:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.83:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.83:6789/0",
              "public_addr": "172.16.12.83:6789/0"
          },
          {
              "rank": 1,
              "name": "amon1",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.81:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.81:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.81:6789/0",
              "public_addr": "172.16.12.81:6789/0"
          },
          {
              "rank": 2,
              "name": "amon2",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.82:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.82:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.82:6789/0",
              "public_addr": "172.16.12.82:6789/0"
          }
      ]
  },
  "feature_map": {
      "mon": [
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 1
          }
      ],
      "osd": [
          {
              "features": "0x3ffddff8ffacffff",
              "release": "luminous",
              "num": 18
          },
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 11
          }
      ],
      "client": [
          {
              "features": "0x3ffddff8ffacffff",
              "release": "luminous",
              "num": 30
          },
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 7
          }
      ],
      "mgr": [
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 2
          }
      ]
  }
}

###########

ceph --admin-daemon /var/run/ceph/ceph-mon.amon1.asok mon_status
{
  "name": "amon1",
  "rank": 1,
  "state": "leader",
  "election_epoch": 4612,
  "quorum": [
      1,
      2
  ],
  "quorum_age": 65744,
  "features": {
      "required_con": "2449958747315912708",
      "required_mon": [
          "kraken",
          "luminous",
          "mimic",
          "osdmap-prune",
          "nautilus"
      ],
      "quorum_con": "4611087854035861503",
      "quorum_mon": [
          "kraken",
          "luminous",
          "mimic",
          "osdmap-prune",
          "nautilus"
      ]
  },
  "outside_quorum": [],
  "extra_probe_peers": [
      {
          "addrvec": [
              {
                  "type": "v2",
                  "addr": "172.16.12.82:3300",
                  "nonce": 0
              },
              {
                  "type": "v1",
                  "addr": "172.16.12.82:6789",
                  "nonce": 0
              }
          ]
      }
  ],
  "sync_provider": [],
  "monmap": {
      "epoch": 14,
      "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
      "modified": "2021-06-22 13:35:56.211202",
      "created": "2018-07-10 12:37:03.124657",
      "min_mon_release": 14,
      "min_mon_release_name": "nautilus",
      "features": {
          "persistent": [
              "kraken",
              "luminous",
              "mimic",
              "osdmap-prune",
              "nautilus"
          ],
          "optional": []
      },
      "mons": [
          {
              "rank": 0,
              "name": "amon3",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.83:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.83:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.83:6789/0",
              "public_addr": "172.16.12.83:6789/0"
          },
          {
              "rank": 1,
              "name": "amon1",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.81:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.81:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.81:6789/0",
              "public_addr": "172.16.12.81:6789/0"
          },
          {
              "rank": 2,
              "name": "amon2",
              "public_addrs": {
                  "addrvec": [
                      {
                          "type": "v2",
                          "addr": "172.16.12.82:3300",
                          "nonce": 0
                      },
                      {
                          "type": "v1",
                          "addr": "172.16.12.82:6789",
                          "nonce": 0
                      }
                  ]
              },
              "addr": "172.16.12.82:6789/0",
              "public_addr": "172.16.12.82:6789/0"
          }
      ]
  },
  "feature_map": {
      "mon": [
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 1
          }
      ],
      "osd": [
          {
              "features": "0x3ffddff8ffacffff",
              "release": "luminous",
              "num": 39
          },
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 8
          }
      ],
      "client": [
          {
              "features": "0x3ffddff8ffacfffb",
              "release": "luminous",
              "num": 1
          },
          {
              "features": "0x3ffddff8ffacffff",
              "release": "luminous",
              "num": 32
          },
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 12
          }
      ],
      "mgr": [
          {
              "features": "0x3ffddff8ffecffff",
              "release": "luminous",
              "num": 1
          }
      ]
  }
}

########
Amon3 log
2021-09-10 12:06:16.409 7f1ead2033c0 0 starting mon.amon3 rank 0 at public addrs [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] at bind addrs [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] mon_data /var/lib/ceph/mon/ceph-amon3 fsid 10a40588-d89e-4fac-be88-afcedc1a7372 2021-09-10 12:06:16.413 7f1ead2033c0 1 mon.amon3@-1(???) e14 preinit fsid 10a40588-d89e-4fac-be88-afcedc1a7372 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 check_fsid cluster_uuid contains '10a40588-d89e-4fac-be88-afcedc1a7372' 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?),4=support erasure code pools,5=new-style osdmap encoding,6=support isa/lrc erasure code,7=support shec erasure code,8=support monmap features,9=luminous ondisk layout,10=mimic ondisk layout,11=nautilus ondisk layout} 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 calc_quorum_requirements required_features 2449958747315912708 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 required_features 2449958747315912708 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 has_ever_joined = 1 2021-09-10 12:06:16.413 7f1ead2033c0 1 mon.amon3@-1(???) e14 preinit clean up potentially inconsistent store state 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 sync_last_committed_floor 353360965 
2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 init_paxos
2021-09-10 12:06:16.413 7f1ead2033c0 5 mon.amon3@-1(???).mds e0 Unable to load 'last_metadata' 
2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).health init
2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).config init
2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 refresh_from_paxos 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).log v0 update_from_paxos 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).log v0 update_from_paxos version 0 summary v 0 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).auth v0 update_from_paxos 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).config load_config got 0 keys 
2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).mgrstat  0
2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).mgrstat check_subs 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).health update_from_paxos 2021-09-10 12:06:16.413 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] learned_addr learned my addr [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] (peer_addr_for_me v2:172.16.12.83:3300/0) 2021-09-10 12:06:16.413 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] _finish_bind bind my_addrs is [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] 2021-09-10 12:06:16.413 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] start start 
2021-09-10 12:06:16.413 7f1ead2033c0  1 --  start start
2021-09-10 12:06:16.413 7f1ead2033c0  2 mon.amon3@-1(???) e14 init
2021-09-10 12:06:16.417 7f1ead2033c0  1  Processor -- start
2021-09-10 12:06:16.417 7f1ead2033c0  1  Processor -- start
2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(???).mgr e0 prime_mgr_client 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 bootstrap 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 sync_reset_requester 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 unregister_cluster_logger - not registered 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 cancel_probe_timeout (none scheduled) 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 monmap e14: 3 mons at {amon1=[v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0],amon2=[v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0],amon3=[v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]} 2021-09-10 12:06:16.417 7f1ead2033c0 0 mon.amon3@-1(probing) e14 my rank is now 0 (was -1) 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] shutdown_connections 
2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 _reset
2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing).auth v0 _set_mon_num_rank num 0 rank 0 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled) 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 timecheck_finish 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 scrub_event_cancel 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 scrub_reset 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled) 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 reset_probe_timeout 0x55ffa1907860 after 2 seconds 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 probing other monitors 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] send_to--> mon [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- ?+0 0x55ffa0956a00 2021-09-10 12:06:16.417 7f1ead2033c0 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] --> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- 0x55ffa0956a00 con 0x55ffa096f600 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] send_to--> mon [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- ?+0 0x55ffa0956780 2021-09-10 12:06:16.417 7f1ead2033c0 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] --> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- 0x55ffa0956780 con 0x55ffa096fa80 2021-09-10 12:06:16.417 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:16.417 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:16.417 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34 2021-09-10 12:06:16.417 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:16.417 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:16.417 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-09-10 12:06:16.421 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:16.421 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:16.425 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 37 2021-09-10 12:06:16.425 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:16.425 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:16.425 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-09-10 12:06:16.617 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:16.617 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:16.621 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34 2021-09-10 12:06:16.621 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:16.621 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:16.621 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-09-10 12:06:16.625 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:16.625 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:16.625 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34 2021-09-10 12:06:16.625 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:16.625 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:16.625 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-09-10 12:06:17.021 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:17.021 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:17.021 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34 2021-09-10 12:06:17.021 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:17.021 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:17.021 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-09-10 12:06:17.029 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:17.029 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:17.029 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34 2021-09-10 12:06:17.029 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:17.029 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:17.029 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-09-10 12:06:17.613 7f1e94dfa700 1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c000 0x55ffa1944000 :6789 s=ACCEPTING pgs=0 cs=0 l=0).send_server_banner sd=34 legacy v1:172.16.12.83:6789/0 socket_addr v1:172.16.12.83:6789/0 target_addr v1:172.16.12.19:41104/0 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_accept con 0x55ffa193c000 no session 2021-09-10 12:06:17.613 7f1e955fb700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] <== client.? v1:172.16.12.19:0/10663437 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (unknown 900162395 0 0) 0x55ffa194c000 con 0x55ffa193c000 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 _ms_dispatch new session 0x55ffa194c240 MonSession(client.? v1:172.16.12.19:0/10663437 is open , features 0x3ffddff8ffecffff (luminous)) features 0x3ffddff8ffecffff 2021-09-10 12:06:17.613 7f1e955fb700 5 mon.amon3@0(probing) e14 waitlisting message auth(proto 0 30 bytes epoch 0) v1 2021-09-10 12:06:17.613 7f1e94dfa700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 legacy=0x55ffa1944000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 34 2021-09-10 12:06:17.613 7f1e94dfa700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 legacy=0x55ffa1944000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-09-10 12:06:17.613 7f1e94dfa700 1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 0x55ffa1944000 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2021-09-10 12:06:17.613 7f1e94dfa700 1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 0x55ffa1944000 :6789 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c000 v1:172.16.12.19:0/10663437 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 reset/close on session client.? v1:172.16.12.19:0/10663437 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 remove_session 0x55ffa194c240 client.? v1:172.16.12.19:0/10663437 features 0x3ffddff8ffecffff 2021-09-10 12:06:17.613 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c480 0x55ffa15d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).accept 2021-09-10 12:06:17.613 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c480 0x55ffa15d2a00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:17.613 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request con 0x55ffa193c480 (start) method 1 payload 22 2021-09-10 12:06:17.613 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request haven't formed initial quorum, EBUSY 2021-09-10 12:06:17.613 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c480 0x55ffa15d2a00 crc :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c480 - 2021-09-10 12:06:17.621 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c900 0x55ffa15d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).accept 2021-09-10 12:06:17.621 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c900 0x55ffa15d3100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:17.621 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request con 0x55ffa193c900 (start) method 1 payload 22 2021-09-10 12:06:17.621 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request haven't formed initial quorum, EBUSY 2021-09-10 12:06:17.621 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c900 0x55ffa15d3100 crc :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-09-10 12:06:17.621 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c900 - 2021-09-10 12:06:17.825 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:17.825 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:17.825 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34 2021-09-10 12:06:17.825 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:17.825 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:17.825 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-09-10 12:06:17.829 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-09-10 12:06:17.829 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon 2021-09-10 12:06:17.829 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34 2021-09-10 12:06:17.829 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-09-10 12:06:17.829 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-09-10 12:06:17.829 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-09-10 12:06:18.417 7f1e97e00700 4 mon.amon3@0(probing) e14 probe_timeout 0x55ffa1907860 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 bootstrap 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 sync_reset_requester 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 unregister_cluster_logger - not registered 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled) 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 monmap e14: 3 mons at {amon1=[v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0],amon2=[v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0],amon3=[v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]} 
2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 _reset
2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing).auth v0 _set_mon_num_rank num 0 rank 0 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled) 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 timecheck_finish 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 scrub_event_cancel 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 scrub_reset 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled) 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 reset_probe_timeout 0x55ffa1907890 after 2 seconds 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 probing other monitors 
BR
Max
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx





_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx





_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx





_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux