I have just seen that on failed mon store.db size is 50K but on both other healthy mons 151M,
What is the best practice? redeploy failed mon?
> On 10. Sep 2021, at 13:08, Eugen Block <eblock@xxxxxx> wrote:
>
> Yes, give it a try. If the cluster is healthy otherwise it shouldn't be a problem.
>
>
> Zitat von mk <mk@xxxxxx>:
>
>> Thx Eugen,
>> just stopping mon and remove/rename only store.db and start mon?
>> BR
>> Max
>>> On 10. Sep 2021, at 12:50, Eugen Block <eblock@xxxxxx> wrote:
>>>
>>> I don't have an explanation but removing the mon store from the failed mon has resolved similar issues in the past. Could you give that a try?
>>>
>>>
>>> Zitat von mk <mk@xxxxxx>:
>>>
>>>> Hi CephFolks,
>>>>
>>>> I have a cluster 14.2.21-22/Ubuntu 18.04 with 3 mon’s. After going down/restart of 1 mon(amon3) it stucks on probing and its also out of quorum, we have changed nothing and it was working, regardless we have checked tcp ports / mtu's of mons are open and reachable.
>>>> appreciate any help you can provide.
>>>>
>>>> ceph health detail
>>>> HEALTH_WARN 1/3 mons down, quorum amon1,amon2
>>>> MON_DOWN 1/3 mons down, quorum amon1,amon2
>>>> mon.amon3 (rank 0) addr [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] is down (out of quorum)
>>>>
>>>> all mon daemons are up and running, monmaptool printout is also equal on all mon nodes and on other healthy mon nodes log there are entries from amon3:
>>>>
>>>> Log from healthy amon1 log:
>>>>
>>>> 2021-09-10 10:49:16.657 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/245515484' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>>>> 2021-09-10 11:04:16.586 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/2472330429' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>>>> 2021-09-10 11:06:06.245 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/1444461156' entity='client.admin' cmd=[{"prefix": "mon getmap"}]: dispatch
>>>> 2021-09-10 11:19:16.581 7fc8285f3700 0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/964654151' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>>>>
>>>> #########
>>>>
>>>> monmaptool --print /tmp/monmap
>>>> monmaptool: monmap file /tmp/monmap
>>>> epoch 14
>>>> fsid 10a40588-d89e-4fac-be88-afcedc1a7372
>>>> last_changed 2021-06-22 13:35:56.211202
>>>> created 2018-07-10 12:37:03.124657
>>>> min_mon_release 14 (nautilus)
>>>> 0: [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] mon.amon3
>>>> 1: [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] mon.amon1
>>>> 2: [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] mon.amon2
>>>>
>>>> ##########
>>>>
>>>> ceph config get mon.*
>>>> WHO MASK LEVEL OPTION VALUE RO
>>>> mon advanced mon_crush_min_required_version firefly *
>>>> mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false
>>>>
>>>> ##########
>>>>
>>>> ceph --admin-daemon /var/run/ceph/ceph-mon.amon3.asok mon_status
>>>> {
>>>> "name": "amon3",
>>>> "rank": 0,
>>>> "state": "probing",
>>>> "election_epoch": 0,
>>>> "quorum": [],
>>>> "features": {
>>>> "required_con": "2449958747315912708",
>>>> "required_mon": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ],
>>>> "quorum_con": "0",
>>>> "quorum_mon": []
>>>> },
>>>> "outside_quorum": [
>>>> "amon3"
>>>> ],
>>>> "extra_probe_peers": [],
>>>> "sync_provider": [],
>>>> "monmap": {
>>>> "epoch": 14,
>>>> "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
>>>> "modified": "2021-06-22 13:35:56.211202",
>>>> "created": "2018-07-10 12:37:03.124657",
>>>> "min_mon_release": 14,
>>>> "min_mon_release_name": "nautilus",
>>>> "features": {
>>>> "persistent": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ],
>>>> "optional": []
>>>> },
>>>> "mons": [
>>>> {
>>>> "rank": 0,
>>>> "name": "amon3",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.83:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.83:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.83:6789/0",
>>>> "public_addr": "172.16.12.83:6789/0"
>>>> },
>>>> {
>>>> "rank": 1,
>>>> "name": "amon1",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.81:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.81:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.81:6789/0",
>>>> "public_addr": "172.16.12.81:6789/0"
>>>> },
>>>> {
>>>> "rank": 2,
>>>> "name": "amon2",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.82:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.82:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.82:6789/0",
>>>> "public_addr": "172.16.12.82:6789/0"
>>>> }
>>>> ]
>>>> },
>>>> "feature_map": {
>>>> "mon": [
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 1
>>>> }
>>>> ]
>>>> }
>>>> }
>>>>
>>>> #########
>>>>
>>>>
>>>> ceph --admin-daemon /var/run/ceph/ceph-mon.amon2.asok mon_status
>>>> {
>>>> "name": "amon2",
>>>> "rank": 2,
>>>> "state": "peon",
>>>> "election_epoch": 4612,
>>>> "quorum": [
>>>> 1,
>>>> 2
>>>> ],
>>>> "quorum_age": 65650,
>>>> "features": {
>>>> "required_con": "2449958747315912708",
>>>> "required_mon": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ],
>>>> "quorum_con": "4611087854035861503",
>>>> "quorum_mon": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ]
>>>> },
>>>> "outside_quorum": [],
>>>> "extra_probe_peers": [],
>>>> "sync_provider": [],
>>>> "monmap": {
>>>> "epoch": 14,
>>>> "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
>>>> "modified": "2021-06-22 11:35:56.211202",
>>>> "created": "2018-07-10 10:37:03.124657",
>>>> "min_mon_release": 14,
>>>> "min_mon_release_name": "nautilus",
>>>> "features": {
>>>> "persistent": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ],
>>>> "optional": []
>>>> },
>>>> "mons": [
>>>> {
>>>> "rank": 0,
>>>> "name": "amon3",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.83:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.83:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.83:6789/0",
>>>> "public_addr": "172.16.12.83:6789/0"
>>>> },
>>>> {
>>>> "rank": 1,
>>>> "name": "amon1",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.81:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.81:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.81:6789/0",
>>>> "public_addr": "172.16.12.81:6789/0"
>>>> },
>>>> {
>>>> "rank": 2,
>>>> "name": "amon2",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.82:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.82:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.82:6789/0",
>>>> "public_addr": "172.16.12.82:6789/0"
>>>> }
>>>> ]
>>>> },
>>>> "feature_map": {
>>>> "mon": [
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 1
>>>> }
>>>> ],
>>>> "osd": [
>>>> {
>>>> "features": "0x3ffddff8ffacffff",
>>>> "release": "luminous",
>>>> "num": 18
>>>> },
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 11
>>>> }
>>>> ],
>>>> "client": [
>>>> {
>>>> "features": "0x3ffddff8ffacffff",
>>>> "release": "luminous",
>>>> "num": 30
>>>> },
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 7
>>>> }
>>>> ],
>>>> "mgr": [
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 2
>>>> }
>>>> ]
>>>> }
>>>> }
>>>>
>>>> ###########
>>>>
>>>> ceph --admin-daemon /var/run/ceph/ceph-mon.amon1.asok mon_status
>>>> {
>>>> "name": "amon1",
>>>> "rank": 1,
>>>> "state": "leader",
>>>> "election_epoch": 4612,
>>>> "quorum": [
>>>> 1,
>>>> 2
>>>> ],
>>>> "quorum_age": 65744,
>>>> "features": {
>>>> "required_con": "2449958747315912708",
>>>> "required_mon": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ],
>>>> "quorum_con": "4611087854035861503",
>>>> "quorum_mon": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ]
>>>> },
>>>> "outside_quorum": [],
>>>> "extra_probe_peers": [
>>>> {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.82:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.82:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> }
>>>> ],
>>>> "sync_provider": [],
>>>> "monmap": {
>>>> "epoch": 14,
>>>> "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
>>>> "modified": "2021-06-22 13:35:56.211202",
>>>> "created": "2018-07-10 12:37:03.124657",
>>>> "min_mon_release": 14,
>>>> "min_mon_release_name": "nautilus",
>>>> "features": {
>>>> "persistent": [
>>>> "kraken",
>>>> "luminous",
>>>> "mimic",
>>>> "osdmap-prune",
>>>> "nautilus"
>>>> ],
>>>> "optional": []
>>>> },
>>>> "mons": [
>>>> {
>>>> "rank": 0,
>>>> "name": "amon3",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.83:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.83:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.83:6789/0",
>>>> "public_addr": "172.16.12.83:6789/0"
>>>> },
>>>> {
>>>> "rank": 1,
>>>> "name": "amon1",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.81:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.81:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.81:6789/0",
>>>> "public_addr": "172.16.12.81:6789/0"
>>>> },
>>>> {
>>>> "rank": 2,
>>>> "name": "amon2",
>>>> "public_addrs": {
>>>> "addrvec": [
>>>> {
>>>> "type": "v2",
>>>> "addr": "172.16.12.82:3300",
>>>> "nonce": 0
>>>> },
>>>> {
>>>> "type": "v1",
>>>> "addr": "172.16.12.82:6789",
>>>> "nonce": 0
>>>> }
>>>> ]
>>>> },
>>>> "addr": "172.16.12.82:6789/0",
>>>> "public_addr": "172.16.12.82:6789/0"
>>>> }
>>>> ]
>>>> },
>>>> "feature_map": {
>>>> "mon": [
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 1
>>>> }
>>>> ],
>>>> "osd": [
>>>> {
>>>> "features": "0x3ffddff8ffacffff",
>>>> "release": "luminous",
>>>> "num": 39
>>>> },
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 8
>>>> }
>>>> ],
>>>> "client": [
>>>> {
>>>> "features": "0x3ffddff8ffacfffb",
>>>> "release": "luminous",
>>>> "num": 1
>>>> },
>>>> {
>>>> "features": "0x3ffddff8ffacffff",
>>>> "release": "luminous",
>>>> "num": 32
>>>> },
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 12
>>>> }
>>>> ],
>>>> "mgr": [
>>>> {
>>>> "features": "0x3ffddff8ffecffff",
>>>> "release": "luminous",
>>>> "num": 1
>>>> }
>>>> ]
>>>> }
>>>> }
>>>>
>>>> ########
>>>> Amon3 log
>>>>
>>>> 2021-09-10 12:06:16.409 7f1ead2033c0 0 starting mon.amon3 rank 0 at public addrs [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] at bind addrs [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] mon_data /var/lib/ceph/mon/ceph-amon3 fsid 10a40588-d89e-4fac-be88-afcedc1a7372
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 1 mon.amon3@-1(???) e14 preinit fsid 10a40588-d89e-4fac-be88-afcedc1a7372
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 check_fsid cluster_uuid contains '10a40588-d89e-4fac-be88-afcedc1a7372'
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?),4=support erasure code pools,5=new-style osdmap encoding,6=support isa/lrc erasure code,7=support shec erasure code,8=support monmap features,9=luminous ondisk layout,10=mimic ondisk layout,11=nautilus ondisk layout}
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 calc_quorum_requirements required_features 2449958747315912708
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 required_features 2449958747315912708
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 has_ever_joined = 1
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 1 mon.amon3@-1(???) e14 preinit clean up potentially inconsistent store state
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 sync_last_committed_floor 353360965
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 init_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 5 mon.amon3@-1(???).mds e0 Unable to load 'last_metadata'
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).health init
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).config init
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 refresh_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).log v0 update_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).log v0 update_from_paxos version 0 summary v 0
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).auth v0 update_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).config load_config got 0 keys
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).mgrstat 0
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).mgrstat check_subs
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).health update_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] learned_addr learned my addr [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] (peer_addr_for_me v2:172.16.12.83:3300/0)
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] _finish_bind bind my_addrs is [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] start start
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 1 -- start start
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 2 mon.amon3@-1(???) e14 init
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 Processor -- start
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 Processor -- start
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(???).mgr e0 prime_mgr_client
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 bootstrap
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 sync_reset_requester
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 unregister_cluster_logger - not registered
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 monmap e14: 3 mons at {amon1=[v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0],amon2=[v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0],amon3=[v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]}
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 0 mon.amon3@-1(probing) e14 my rank is now 0 (was -1)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] shutdown_connections
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 _reset
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing).auth v0 _set_mon_num_rank num 0 rank 0
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 timecheck_finish
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 scrub_event_cancel
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 scrub_reset
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 reset_probe_timeout 0x55ffa1907860 after 2 seconds
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 probing other monitors
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] send_to--> mon [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- ?+0 0x55ffa0956a00
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] --> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- 0x55ffa0956a00 con 0x55ffa096f600
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] send_to--> mon [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- ?+0 0x55ffa0956780
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] --> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- 0x55ffa0956780 con 0x55ffa096fa80
>>>> 2021-09-10 12:06:16.417 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.417 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.417 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:16.417 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.417 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.417 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000
>>>> 2021-09-10 12:06:16.421 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.421 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.425 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 37
>>>> 2021-09-10 12:06:16.425 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.425 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.425 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000
>>>> 2021-09-10 12:06:16.617 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.617 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.621 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:16.621 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.621 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.621 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000
>>>> 2021-09-10 12:06:16.625 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.625 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.625 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:16.625 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.625 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.625 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000
>>>> 2021-09-10 12:06:17.021 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.021 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.021 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.021 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.021 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.021 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000
>>>> 2021-09-10 12:06:17.029 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.029 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.029 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.029 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.029 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.029 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c000 0x55ffa1944000 :6789 s=ACCEPTING pgs=0 cs=0 l=0).send_server_banner sd=34 legacy v1:172.16.12.83:6789/0 socket_addr v1:172.16.12.83:6789/0 target_addr v1:172.16.12.19:41104/0
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_accept con 0x55ffa193c000 no session
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] <== client.? v1:172.16.12.19:0/10663437 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (unknown 900162395 0 0) 0x55ffa194c000 con 0x55ffa193c000
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 _ms_dispatch new session 0x55ffa194c240 MonSession(client.? v1:172.16.12.19:0/10663437 is open , features 0x3ffddff8ffecffff (luminous)) features 0x3ffddff8ffecffff
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 5 mon.amon3@0(probing) e14 waitlisting message auth(proto 0 30 bytes epoch 0) v1
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 legacy=0x55ffa1944000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 legacy=0x55ffa1944000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 0x55ffa1944000 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 0x55ffa1944000 :6789 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c000 v1:172.16.12.19:0/10663437
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 reset/close on session client.? v1:172.16.12.19:0/10663437
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 remove_session 0x55ffa194c240 client.? v1:172.16.12.19:0/10663437 features 0x3ffddff8ffecffff
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c480 0x55ffa15d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).accept
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c480 0x55ffa15d2a00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request con 0x55ffa193c480 (start) method 1 payload 22
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request haven't formed initial quorum, EBUSY
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c480 0x55ffa15d2a00 crc :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c480 -
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c900 0x55ffa15d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).accept
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c900 0x55ffa15d3100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request con 0x55ffa193c900 (start) method 1 payload 22
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request haven't formed initial quorum, EBUSY
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> conn(0x55ffa193c900 0x55ffa15d3100 crc :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop
>>>> 2021-09-10 12:06:17.621 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c900 -
>>>> 2021-09-10 12:06:17.825 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.825 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.825 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.825 7f1e99e04700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.825 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.825 7f1e99e04700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000
>>>> 2021-09-10 12:06:17.829 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.829 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.829 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.829 7f1e99603700 1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.829 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.829 7f1e99603700 1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 4 mon.amon3@0(probing) e14 probe_timeout 0x55ffa1907860
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 bootstrap
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 sync_reset_requester
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 unregister_cluster_logger - not registered
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 monmap e14: 3 mons at {amon1=[v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0],amon2=[v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0],amon3=[v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]}
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 _reset
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing).auth v0 _set_mon_num_rank num 0 rank 0
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 timecheck_finish
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 scrub_event_cancel
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 scrub_reset
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 reset_probe_timeout 0x55ffa1907890 after 2 seconds
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 probing other monitors
>>>> BR
>>>> Max
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
