Re: mon stucks on probing and out of quorum, after down and restart

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



no doesn’t start the mon daemon:
Sep 10 13:35:55 amon3 systemd[1]: ceph-mon@amon3.service: Start request repeated too quickly.
Sep 10 13:35:55 amon3 systemd[1]: ceph-mon@amon3.service: Failed with result 'exit-code'.
Sep 10 13:35:55 amon3 systemd[1]: Failed to start Ceph cluster monitor daemon.

> On 10. Sep 2021, at 13:08, Eugen Block <eblock@xxxxxx> wrote:
> 
> Yes, give it a try. If the cluster is healthy otherwise it shouldn't be a problem.
> 
> 
> Zitat von mk <mk@xxxxxx>:
> 
>> Thx Eugen,
>> just stopping mon and remove/rename only store.db and start mon?
>> BR
>> Max
>>> On 10. Sep 2021, at 12:50, Eugen Block <eblock@xxxxxx> wrote:
>>> 
>>> I don't have an explanation but removing the mon store from the failed mon has resolved similar issues in the past. Could you give that a try?
>>> 
>>> 
>>> Zitat von mk <mk@xxxxxx>:
>>> 
>>>> Hi CephFolks,
>>>> 
>>>> I have a cluster 14.2.21-22/Ubuntu 18.04 with 3 mon’s. After going down/restart of 1 mon(amon3) it stucks on probing and its also out of quorum, we have changed nothing and it was working, regardless we have checked tcp ports / mtu's of mons are open and reachable.
>>>> appreciate any help you can provide.
>>>> 
>>>> ceph health detail
>>>> HEALTH_WARN 1/3 mons down, quorum amon1,amon2
>>>> MON_DOWN 1/3 mons down, quorum amon1,amon2
>>>>   mon.amon3 (rank 0) addr [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] is down (out of quorum)
>>>> 
>>>> all mon daemons are up and running, monmaptool printout is also equal on all mon nodes and on other healthy mon nodes log there are entries from amon3:
>>>> 
>>>> Log from healthy amon1 log:
>>>> 
>>>> 2021-09-10 10:49:16.657 7fc8285f3700  0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/245515484' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>>>> 2021-09-10 11:04:16.586 7fc8285f3700  0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/2472330429' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>>>> 2021-09-10 11:06:06.245 7fc8285f3700  0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/1444461156' entity='client.admin' cmd=[{"prefix": "mon getmap"}]: dispatch
>>>> 2021-09-10 11:19:16.581 7fc8285f3700  0 log_channel(audit) log [DBG] : from='client.? 172.16.12.83:0/964654151' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>>>> 
>>>> #########
>>>> 
>>>> monmaptool --print /tmp/monmap
>>>> monmaptool: monmap file /tmp/monmap
>>>> epoch 14
>>>> fsid 10a40588-d89e-4fac-be88-afcedc1a7372
>>>> last_changed 2021-06-22 13:35:56.211202
>>>> created 2018-07-10 12:37:03.124657
>>>> min_mon_release 14 (nautilus)
>>>> 0: [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] mon.amon3
>>>> 1: [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] mon.amon1
>>>> 2: [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] mon.amon2
>>>> 
>>>> ##########
>>>> 
>>>> ceph config get mon.*
>>>> WHO MASK LEVEL    OPTION                                         VALUE   RO
>>>> mon      advanced mon_crush_min_required_version                 firefly *
>>>> mon      advanced mon_warn_on_insecure_global_id_reclaim_allowed false
>>>> 
>>>> ##########
>>>> 
>>>> ceph --admin-daemon /var/run/ceph/ceph-mon.amon3.asok mon_status
>>>> {
>>>>   "name": "amon3",
>>>>   "rank": 0,
>>>>   "state": "probing",
>>>>   "election_epoch": 0,
>>>>   "quorum": [],
>>>>   "features": {
>>>>       "required_con": "2449958747315912708",
>>>>       "required_mon": [
>>>>           "kraken",
>>>>           "luminous",
>>>>           "mimic",
>>>>           "osdmap-prune",
>>>>           "nautilus"
>>>>       ],
>>>>       "quorum_con": "0",
>>>>       "quorum_mon": []
>>>>   },
>>>>   "outside_quorum": [
>>>>       "amon3"
>>>>   ],
>>>>   "extra_probe_peers": [],
>>>>   "sync_provider": [],
>>>>   "monmap": {
>>>>       "epoch": 14,
>>>>       "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
>>>>       "modified": "2021-06-22 13:35:56.211202",
>>>>       "created": "2018-07-10 12:37:03.124657",
>>>>       "min_mon_release": 14,
>>>>       "min_mon_release_name": "nautilus",
>>>>       "features": {
>>>>           "persistent": [
>>>>               "kraken",
>>>>               "luminous",
>>>>               "mimic",
>>>>               "osdmap-prune",
>>>>               "nautilus"
>>>>           ],
>>>>           "optional": []
>>>>       },
>>>>       "mons": [
>>>>           {
>>>>               "rank": 0,
>>>>               "name": "amon3",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.83:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.83:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.83:6789/0",
>>>>               "public_addr": "172.16.12.83:6789/0"
>>>>           },
>>>>           {
>>>>               "rank": 1,
>>>>               "name": "amon1",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.81:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.81:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.81:6789/0",
>>>>               "public_addr": "172.16.12.81:6789/0"
>>>>           },
>>>>           {
>>>>               "rank": 2,
>>>>               "name": "amon2",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.82:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.82:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.82:6789/0",
>>>>               "public_addr": "172.16.12.82:6789/0"
>>>>           }
>>>>       ]
>>>>   },
>>>>   "feature_map": {
>>>>       "mon": [
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 1
>>>>           }
>>>>       ]
>>>>   }
>>>> }
>>>> 
>>>> #########
>>>> 
>>>> 
>>>> ceph --admin-daemon /var/run/ceph/ceph-mon.amon2.asok mon_status
>>>> {
>>>>   "name": "amon2",
>>>>   "rank": 2,
>>>>   "state": "peon",
>>>>   "election_epoch": 4612,
>>>>   "quorum": [
>>>>       1,
>>>>       2
>>>>   ],
>>>>   "quorum_age": 65650,
>>>>   "features": {
>>>>       "required_con": "2449958747315912708",
>>>>       "required_mon": [
>>>>           "kraken",
>>>>           "luminous",
>>>>           "mimic",
>>>>           "osdmap-prune",
>>>>           "nautilus"
>>>>       ],
>>>>       "quorum_con": "4611087854035861503",
>>>>       "quorum_mon": [
>>>>           "kraken",
>>>>           "luminous",
>>>>           "mimic",
>>>>           "osdmap-prune",
>>>>           "nautilus"
>>>>       ]
>>>>   },
>>>>   "outside_quorum": [],
>>>>   "extra_probe_peers": [],
>>>>   "sync_provider": [],
>>>>   "monmap": {
>>>>       "epoch": 14,
>>>>       "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
>>>>       "modified": "2021-06-22 11:35:56.211202",
>>>>       "created": "2018-07-10 10:37:03.124657",
>>>>       "min_mon_release": 14,
>>>>       "min_mon_release_name": "nautilus",
>>>>       "features": {
>>>>           "persistent": [
>>>>               "kraken",
>>>>               "luminous",
>>>>               "mimic",
>>>>               "osdmap-prune",
>>>>               "nautilus"
>>>>           ],
>>>>           "optional": []
>>>>       },
>>>>       "mons": [
>>>>           {
>>>>               "rank": 0,
>>>>               "name": "amon3",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.83:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.83:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.83:6789/0",
>>>>               "public_addr": "172.16.12.83:6789/0"
>>>>           },
>>>>           {
>>>>               "rank": 1,
>>>>               "name": "amon1",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.81:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.81:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.81:6789/0",
>>>>               "public_addr": "172.16.12.81:6789/0"
>>>>           },
>>>>           {
>>>>               "rank": 2,
>>>>               "name": "amon2",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.82:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.82:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.82:6789/0",
>>>>               "public_addr": "172.16.12.82:6789/0"
>>>>           }
>>>>       ]
>>>>   },
>>>>   "feature_map": {
>>>>       "mon": [
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 1
>>>>           }
>>>>       ],
>>>>       "osd": [
>>>>           {
>>>>               "features": "0x3ffddff8ffacffff",
>>>>               "release": "luminous",
>>>>               "num": 18
>>>>           },
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 11
>>>>           }
>>>>       ],
>>>>       "client": [
>>>>           {
>>>>               "features": "0x3ffddff8ffacffff",
>>>>               "release": "luminous",
>>>>               "num": 30
>>>>           },
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 7
>>>>           }
>>>>       ],
>>>>       "mgr": [
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 2
>>>>           }
>>>>       ]
>>>>   }
>>>> }
>>>> 
>>>> ###########
>>>> 
>>>> ceph --admin-daemon /var/run/ceph/ceph-mon.amon1.asok mon_status
>>>> {
>>>>   "name": "amon1",
>>>>   "rank": 1,
>>>>   "state": "leader",
>>>>   "election_epoch": 4612,
>>>>   "quorum": [
>>>>       1,
>>>>       2
>>>>   ],
>>>>   "quorum_age": 65744,
>>>>   "features": {
>>>>       "required_con": "2449958747315912708",
>>>>       "required_mon": [
>>>>           "kraken",
>>>>           "luminous",
>>>>           "mimic",
>>>>           "osdmap-prune",
>>>>           "nautilus"
>>>>       ],
>>>>       "quorum_con": "4611087854035861503",
>>>>       "quorum_mon": [
>>>>           "kraken",
>>>>           "luminous",
>>>>           "mimic",
>>>>           "osdmap-prune",
>>>>           "nautilus"
>>>>       ]
>>>>   },
>>>>   "outside_quorum": [],
>>>>   "extra_probe_peers": [
>>>>       {
>>>>           "addrvec": [
>>>>               {
>>>>                   "type": "v2",
>>>>                   "addr": "172.16.12.82:3300",
>>>>                   "nonce": 0
>>>>               },
>>>>               {
>>>>                   "type": "v1",
>>>>                   "addr": "172.16.12.82:6789",
>>>>                   "nonce": 0
>>>>               }
>>>>           ]
>>>>       }
>>>>   ],
>>>>   "sync_provider": [],
>>>>   "monmap": {
>>>>       "epoch": 14,
>>>>       "fsid": "10a40588-d89e-4fac-be88-afcedc1a7372",
>>>>       "modified": "2021-06-22 13:35:56.211202",
>>>>       "created": "2018-07-10 12:37:03.124657",
>>>>       "min_mon_release": 14,
>>>>       "min_mon_release_name": "nautilus",
>>>>       "features": {
>>>>           "persistent": [
>>>>               "kraken",
>>>>               "luminous",
>>>>               "mimic",
>>>>               "osdmap-prune",
>>>>               "nautilus"
>>>>           ],
>>>>           "optional": []
>>>>       },
>>>>       "mons": [
>>>>           {
>>>>               "rank": 0,
>>>>               "name": "amon3",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.83:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.83:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.83:6789/0",
>>>>               "public_addr": "172.16.12.83:6789/0"
>>>>           },
>>>>           {
>>>>               "rank": 1,
>>>>               "name": "amon1",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.81:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.81:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.81:6789/0",
>>>>               "public_addr": "172.16.12.81:6789/0"
>>>>           },
>>>>           {
>>>>               "rank": 2,
>>>>               "name": "amon2",
>>>>               "public_addrs": {
>>>>                   "addrvec": [
>>>>                       {
>>>>                           "type": "v2",
>>>>                           "addr": "172.16.12.82:3300",
>>>>                           "nonce": 0
>>>>                       },
>>>>                       {
>>>>                           "type": "v1",
>>>>                           "addr": "172.16.12.82:6789",
>>>>                           "nonce": 0
>>>>                       }
>>>>                   ]
>>>>               },
>>>>               "addr": "172.16.12.82:6789/0",
>>>>               "public_addr": "172.16.12.82:6789/0"
>>>>           }
>>>>       ]
>>>>   },
>>>>   "feature_map": {
>>>>       "mon": [
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 1
>>>>           }
>>>>       ],
>>>>       "osd": [
>>>>           {
>>>>               "features": "0x3ffddff8ffacffff",
>>>>               "release": "luminous",
>>>>               "num": 39
>>>>           },
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 8
>>>>           }
>>>>       ],
>>>>       "client": [
>>>>           {
>>>>               "features": "0x3ffddff8ffacfffb",
>>>>               "release": "luminous",
>>>>               "num": 1
>>>>           },
>>>>           {
>>>>               "features": "0x3ffddff8ffacffff",
>>>>               "release": "luminous",
>>>>               "num": 32
>>>>           },
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 12
>>>>           }
>>>>       ],
>>>>       "mgr": [
>>>>           {
>>>>               "features": "0x3ffddff8ffecffff",
>>>>               "release": "luminous",
>>>>               "num": 1
>>>>           }
>>>>       ]
>>>>   }
>>>> }
>>>> 
>>>> ########
>>>> Amon3 log
>>>> 
>>>> 2021-09-10 12:06:16.409 7f1ead2033c0  0 starting mon.amon3 rank 0 at public addrs [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] at bind addrs [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] mon_data /var/lib/ceph/mon/ceph-amon3 fsid 10a40588-d89e-4fac-be88-afcedc1a7372
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  1 mon.amon3@-1(???) e14 preinit fsid 10a40588-d89e-4fac-be88-afcedc1a7372
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 check_fsid cluster_uuid contains '10a40588-d89e-4fac-be88-afcedc1a7372'
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?),4=support erasure code pools,5=new-style osdmap encoding,6=support isa/lrc erasure code,7=support shec erasure code,8=support monmap features,9=luminous ondisk layout,10=mimic ondisk layout,11=nautilus ondisk layout}
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 calc_quorum_requirements required_features 2449958747315912708
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 required_features 2449958747315912708
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 has_ever_joined = 1
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  1 mon.amon3@-1(???) e14 preinit clean up potentially inconsistent store state
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 sync_last_committed_floor 353360965
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 init_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  5 mon.amon3@-1(???).mds e0 Unable to load 'last_metadata'
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).health init
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).config init
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???) e14 refresh_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).log v0 update_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).log v0 update_from_paxos version 0 summary v 0
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).auth v0 update_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).config load_config got 0 keys
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).mgrstat  0
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).mgrstat check_subs
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0 10 mon.amon3@-1(???).health update_from_paxos
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] learned_addr learned my addr [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] (peer_addr_for_me v2:172.16.12.83:3300/0)
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] _finish_bind bind my_addrs is [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] start start
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  1 --  start start
>>>> 2021-09-10 12:06:16.413 7f1ead2033c0  2 mon.amon3@-1(???) e14 init
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1  Processor -- start
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1  Processor -- start
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(???).mgr e0 prime_mgr_client
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 bootstrap
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 sync_reset_requester
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 unregister_cluster_logger - not registered
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@-1(probing) e14 monmap e14: 3 mons at {amon1=[v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0],amon2=[v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0],amon3=[v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]}
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  0 mon.amon3@-1(probing) e14  my rank is now 0 (was -1)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] shutdown_connections
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 _reset
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing).auth v0 _set_mon_num_rank num 0 rank 0
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 timecheck_finish
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 scrub_event_cancel
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 scrub_reset
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 reset_probe_timeout 0x55ffa1907860 after 2 seconds
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0 10 mon.amon3@0(probing) e14 probing other monitors
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] send_to--> mon [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- ?+0 0x55ffa0956a00
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] --> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- 0x55ffa0956a00 con 0x55ffa096f600
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] send_to--> mon [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- ?+0 0x55ffa0956780
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect
>>>> 2021-09-10 12:06:16.417 7f1ead2033c0  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] --> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] -- mon_probe(probe 10a40588-d89e-4fac-be88-afcedc1a7372 name amon3 mon_release 14) v7 -- 0x55ffa0956780 con 0x55ffa096fa80
>>>> 2021-09-10 12:06:16.417 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.417 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.417 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:16.417 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.417 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.417 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000
>>>> 2021-09-10 12:06:16.421 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.421 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.425 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 37
>>>> 2021-09-10 12:06:16.425 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.425 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.425 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000
>>>> 2021-09-10 12:06:16.617 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.617 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.621 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:16.621 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.621 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.621 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000
>>>> 2021-09-10 12:06:16.625 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:16.625 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:16.625 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:16.625 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:16.625 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:16.625 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000
>>>> 2021-09-10 12:06:17.021 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.021 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.021 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.021 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.021 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.021 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000
>>>> 2021-09-10 12:06:17.029 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.029 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.029 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.029 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.029 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.029 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >>  conn(0x55ffa193c000 0x55ffa1944000 :6789 s=ACCEPTING pgs=0 cs=0 l=0).send_server_banner sd=34 legacy v1:172.16.12.83:6789/0 socket_addr v1:172.16.12.83:6789/0 target_addr v1:172.16.12.19:41104/0
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_accept con 0x55ffa193c000 no session
>>>> 2021-09-10 12:06:17.613 7f1e955fb700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] <== client.? v1:172.16.12.19:0/10663437 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (unknown 900162395 0 0) 0x55ffa194c000 con 0x55ffa193c000
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 _ms_dispatch new session 0x55ffa194c240 MonSession(client.? v1:172.16.12.19:0/10663437 is open , features 0x3ffddff8ffecffff (luminous)) features 0x3ffddff8ffecffff
>>>> 2021-09-10 12:06:17.613 7f1e955fb700  5 mon.amon3@0(probing) e14 waitlisting message auth(proto 0 30 bytes epoch 0) v1
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 legacy=0x55ffa1944000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 legacy=0x55ffa1944000 unknown :6789 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 0x55ffa1944000 :6789 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 --1- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> v1:172.16.12.19:0/10663437 conn(0x55ffa193c000 0x55ffa1944000 :6789 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c000 v1:172.16.12.19:0/10663437
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 reset/close on session client.? v1:172.16.12.19:0/10663437
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 remove_session 0x55ffa194c240 client.? v1:172.16.12.19:0/10663437 features 0x3ffddff8ffecffff
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >>  conn(0x55ffa193c480 0x55ffa15d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).accept
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >>  conn(0x55ffa193c480 0x55ffa15d2a00 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request con 0x55ffa193c480 (start) method 1 payload 22
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request haven't formed initial quorum, EBUSY
>>>> 2021-09-10 12:06:17.613 7f1e94dfa700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >>  conn(0x55ffa193c480 0x55ffa15d2a00 crc :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop
>>>> 2021-09-10 12:06:17.613 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c480 -
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >>  conn(0x55ffa193c900 0x55ffa15d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).accept
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >>  conn(0x55ffa193c900 0x55ffa15d3100 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request con 0x55ffa193c900 (start) method 1 payload 22
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700 10 mon.amon3@0(probing) e14 handle_auth_request haven't formed initial quorum, EBUSY
>>>> 2021-09-10 12:06:17.621 7f1e94dfa700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >>  conn(0x55ffa193c900 0x55ffa15d3100 crc :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop
>>>> 2021-09-10 12:06:17.621 7f1e955fb700 10 mon.amon3@0(probing) e14 ms_handle_reset 0x55ffa193c900 -
>>>> 2021-09-10 12:06:17.825 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.825 7f1e99e04700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.825 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.825 7f1e99e04700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 msgr2=0x55ffa15d1c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.825 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.825 7f1e99e04700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0] conn(0x55ffa096f600 0x55ffa15d1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000
>>>> 2021-09-10 12:06:17.829 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
>>>> 2021-09-10 12:06:17.829 7f1e99603700 10 mon.amon3@0(probing) e14 ms_get_authorizer for mon
>>>> 2021-09-10 12:06:17.829 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 34
>>>> 2021-09-10 12:06:17.829 7f1e99603700  1 -- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 msgr2=0x55ffa15d2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed
>>>> 2021-09-10 12:06:17.829 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
>>>> 2021-09-10 12:06:17.829 7f1e99603700  1 --2- [v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0] >> [v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0] conn(0x55ffa096fa80 0x55ffa15d2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000
>>>> 2021-09-10 12:06:18.417 7f1e97e00700  4 mon.amon3@0(probing) e14 probe_timeout 0x55ffa1907860
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 bootstrap
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 sync_reset_requester
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 unregister_cluster_logger - not registered
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 monmap e14: 3 mons at {amon1=[v2:172.16.12.81:3300/0,v1:172.16.12.81:6789/0],amon2=[v2:172.16.12.82:3300/0,v1:172.16.12.82:6789/0],amon3=[v2:172.16.12.83:3300/0,v1:172.16.12.83:6789/0]}
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 _reset
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing).auth v0 _set_mon_num_rank num 0 rank 0
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 timecheck_finish
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 scrub_event_cancel
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 scrub_reset
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 cancel_probe_timeout (none scheduled)
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 reset_probe_timeout 0x55ffa1907890 after 2 seconds
>>>> 2021-09-10 12:06:18.417 7f1e97e00700 10 mon.amon3@0(probing) e14 probing other monitors
>>>> BR
>>>> Max
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> 
> 
> 
> 
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx





[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux