got the folowing outputt since ive changed the debug setting:
2018-07-26 11:46:21.004490 7f819e968700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0
2018-07-26 11:46:21.004520 7f81a196e700 10 -- 10.111.73.1:6789/0 dispatch_throttle_release 60 to dispatch throttler 60/104857600
2018-07-26 11:46:23.058057 7f81a4173700 1 -- 10.111.73.1:6789/0 >> 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77 cs=1 l=1).mark_down
2018-07-26 11:46:23.058084 7f81a4173700 2 -- 10.111.73.1:6789/0 >> 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77 cs=1 l=1)._stop
2018-07-26 11:46:23.058094 7f81a4173700 10 -- 10.111.73.1:6789/0 >> 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77 cs=1 l=1).discard_out_queue started
2018-07-26 11:46:23.058120 7f81a4173700 1 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1).mark_down
2018-07-26 11:46:23.058131 7f81a4173700 2 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1)._stop
2018-07-26 11:46:23.058143 7f81a4173700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1).discard_out_queue started
2018-07-26 11:46:23.962796 7f819d966700 10 Processor -- accept listen_fd=22
2018-07-26 11:46:23.962845 7f819d966700 10 Processor -- accept accepted incoming on sd 23
2018-07-26 11:46:23.962858 7f819d966700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=23
2018-07-26 11:46:23.962929 7f819e167700 1 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=23 -
2018-07-26 11:46:23.963022 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0
2018-07-26 11:46:23.963045 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: -
2018-07-26 11:46:23.963091 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection accept peer addr is 10.111.73.1:0/1745436331
2018-07-26 11:46:23.963190 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1)._process_connection accept of host_type 8, policy.lossy=1 policy.server=1 policy.standby=0 policy.resetcheck=0
2018-07-26 11:46:23.963216 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept my proto 15, their proto 15
2018-07-26 11:46:23.963232 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept setting up session_security.
2018-07-26 11:46:23.963248 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept new session
2018-07-26 11:46:23.963256 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1 l=1).handle_connect_msg accept success, connect_seq = 1 in_seq=0, sending READY
2018-07-26 11:46:23.963264 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1 l=1).handle_connect_msg accept features 4611087853745930235
2018-07-26 11:46:23.963315 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1 l=1)._try_send sent bytes 34 remaining bytes 0
2018-07-26 11:46:23.963356 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1).handle_connect_msg accept write reply msg done
2018-07-26 11:46:23.963442 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1)._process_connection accept get newly_acked_seq 0
2018-07-26 11:46:23.963461 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1).discard_requeued_up_to 0
2018-07-26 11:46:23.963634 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_KEEPALIVE2 pgs=87 cs=1 l=1)._append_keepalive_or_ack
2018-07-26 11:46:23.963658 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=87 cs=1 l=1).process wants 60 bytes from policy throttler 120/104857600
2018-07-26 11:46:23.963679 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1).process aborted = 0
2018-07-26 11:46:23.963705 7f819e167700 5 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx client.? seq 1 0x55aa46be4480 auth(proto 0 30 bytes epoch 0) v1
2018-07-26 11:46:23.963750 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN pgs=87 cs=1 l=1).handle_write
2018-07-26 11:46:23.963755 7f81a196e700 1 -- 10.111.73.1:6789/0 <== client.? 10.111.73.1:0/1745436331 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (4135352935 0 0) 0x55aa46be4480 con 0x55aa46afd800
2018-07-26 11:46:23.963808 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN pgs=87 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0
2018-07-26 11:46:23.963823 7f81a196e700 10 -- 10.111.73.1:6789/0 dispatch_throttle_release 60 to dispatch throttler 60/104857600
2018-07-26 11:46:24.003866 7f819d966700 10 Processor -- accept listen_fd=22
2018-07-26 11:46:24.003902 7f819d966700 10 Processor -- accept accepted incoming on sd 26
2018-07-26 11:46:24.003911 7f819d966700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=26
2018-07-26 11:46:24.004001 7f819e167700 1 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=26 -
2018-07-26 11:46:24.004057 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0
2018-07-26 11:46:24.004071 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: -
2018-07-26 11:46:24.004199 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection accept peer addr is 10.111.73.3:0/1033315403
2018-07-26 11:46:24.004286 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1)._process_connection accept of host_type 8, policy.lossy=1 policy.server=1 policy.standby=0 policy.resetcheck=0
2018-07-26 11:46:24.004304 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept my proto 15, their proto 15
2018-07-26 11:46:24.004319 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept setting up session_security.
2018-07-26 11:46:24.004338 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept new session
2018-07-26 11:46:24.004351 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1 l=1).handle_connect_msg accept success, connect_seq = 1 in_seq=0, sending READY
2018-07-26 11:46:24.004365 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1 l=1).handle_connect_msg accept features 4611087853745930235
2018-07-26 11:46:24.004463 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1 l=1)._try_send sent bytes 34 remaining bytes 0
2018-07-26 11:46:24.004489 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1).handle_connect_msg accept write reply msg done
2018-07-26 11:46:24.004634 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1)._process_connection accept get newly_acked_seq 0
2018-07-26 11:46:24.004650 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1).discard_requeued_up_to 0
2018-07-26 11:46:24.004807 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_KEEPALIVE2 pgs=74 cs=1 l=1)._append_keepalive_or_ack
2018-07-26 11:46:24.004828 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=74 cs=1 l=1).process wants 60 bytes from policy throttler 180/104857600
2018-07-26 11:46:24.004847 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74 cs=1 l=1).process aborted = 0
2018-07-26 11:46:24.004873 7f819e167700 5 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74 cs=1 l=1). rx client.? seq 1 0x55aa46be4fc0 auth(proto 0 30 bytes epoch 0) v1
2018-07-26 11:46:24.004914 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN pgs=74 cs=1 l=1).handle_write
2018-07-26 11:46:24.004921 7f81a196e700 1 -- 10.111.73.1:6789/0 <== client.? 10.111.73.3:0/1033315403 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (2547518125 0 0) 0x55aa46be4fc0 con 0x55aa46bc1000
2018-07-26 11:46:24.004954 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN pgs=74 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0
2018-07-26 11:46:24.004965 7f81a196e700 10 -- 10.111.73.1:6789/0 dispatch_throttle_release 60 to dispatch throttler 60/104857600
kind regards
ben
> Wido den Hollander <wido@xxxxxxxx> hat am 26. Juli 2018 um 11:07 geschrieben:
>
>
>
>
> On 07/26/2018 10:33 AM, Benjamin Naber wrote:
> > hi Wido,
> >
> > thx for your reply.
> > time is also in sync. i forced time sync again to be sure.
> >
>
> Try setting debug_mon to 10 or even 20 and check the logs about what the
> MONs are saying.
>
> debug_ms = 10 might also help to get some more information about the
> Messenger Traffic.
>
> Wido
>
> > kind regards
> >
> > Ben
> >
> >> Wido den Hollander <wido@xxxxxxxx> hat am 26. Juli 2018 um 10:18
> > geschrieben:
> >>
> >>
> >>
> >>
> >> On 07/26/2018 10:12 AM, Benjamin Naber wrote:
> >> > Hi together,
> >> >
> >> > we currently have some problems with monitor quorum after shutting
> > down all cluster nodes for migration to another location.
> >> >
> >> > mon_status gives uns the following outputt:
> >> >
> >> > {
> >> > "name": "mon01",
> >> > "rank": 0,
> >> > "state": "electing",
> >> > "election_epoch": 20345,
> >> > "quorum": [],
> >> > "features": {
> >> > "required_con": "153140804152475648",
> >> > "required_mon": [
> >> > "kraken",
> >> > "luminous"
> >> > ],
> >> > "quorum_con": "0",
> >> > "quorum_mon": []
> >> > },
> >> > "outside_quorum": [],
> >> > "extra_probe_peers": [],
> >> > "sync_provider": [],
> >> > "monmap": {
> >> > "epoch": 1,
> >> > "fsid": "c1e3c489-67a4-47a2-a3ca-98816d1c9d44",
> >> > "modified": "2018-06-21 13:48:58.796939",
> >> > "created": "2018-06-21 13:48:58.796939",
> >> > "features": {
> >> > "persistent": [
> >> > "kraken",
> >> > "luminous"
> >> > ],
> >> > "optional": []
> >> > },
> >> > "mons": [
> >> > {
> >> > "rank": 0,
> >> > "name": "mon01",
> >> > "addr": "10.111.73.1:6789/0",
> >> > "public_addr": "10.111.73.1:6789/0"
> >> > },
> >> > {
> >> > "rank": 1,
> >> > "name": "mon02",
> >> > "addr": "10.111.73.2:6789/0",
> >> > "public_addr": "10.111.73.2:6789/0"
> >> > },
> >> > {
> >> > "rank": 2,
> >> > "name": "mon03",
> >> > "addr": "10.111.73.3:6789/0",
> >> > "public_addr": "10.111.73.3:6789/0"
> >> > }
> >> > ]
> >> > },
> >> > "feature_map": {
> >> > "mon": {
> >> > "group": {
> >> > "features": "0x3ffddff8eea4fffb",
> >> > "release": "luminous",
> >> > "num": 1
> >> > }
> >> > }
> >> > }
> >> > }
> >> >
> >> > ceph ping mon.id gives us also just dosent work. monitoring nodes
> > have full network connectivity. firewall rules are also ok.
> >> >
> >> > what cloud be the reson for stucking quorum election ?
> >> >
> >>
> >> Is the time in sync between the nodes?
> >>
> >> Wido
> >>
> >> > kind regards
> >> >
> >> > Ben
> >> > _______________________________________________
> >> > ceph-users mailing list
> >> > ceph-users@xxxxxxxxxxxxxx
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
