I try to upgrade three-node nautilus cluster to pacific. I am updating ceph
on one node and restarting daemons. OSD ok, but monitor cannot enter quorum.
With debug_mon 20/20 I see repeating blocks in the logs of problem monitor
like
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
bootstrap
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
sync_reset_requester
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
unregister_cluster_logger - not registered
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
cancel_probe_timeout 0x557603d82420
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
monmap e4: 3 mons at {debian1=[v2:
172.16.21.101:3300/0,v1:172.16.21.101:6789/0],debian2=[v2:
172.16.21.102:3300/0,v1:172.16.21.102:6789/0],debian3=[v2:
172.16.21.103:3300/0,v1:172.16.21.103:6789/0]}
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
_reset
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing).auth v0
_set_mon_num_rank num 0 rank 0
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
cancel_probe_timeout (none scheduled)
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
timecheck_finish
2021-12-15T13:34:57.075+1000 7f6e1b417700 15 mon.debian2@1(probing) e4
health_tick_stop
2021-12-15T13:34:57.075+1000 7f6e1b417700 15 mon.debian2@1(probing) e4
health_interval_stop
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
scrub_event_cancel
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
scrub_reset
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
cancel_probe_timeout (none scheduled)
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
reset_probe_timeout 0x557603d82420 after 2 seconds
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
probing other monitors
2021-12-15T13:34:57.075+1000 7f6e1b417700 20 mon.debian2@1(probing) e4
_ms_dispatch existing session 0x557603d60b40 for mon.2
2021-12-15T13:34:57.075+1000 7f6e1b417700 20 mon.debian2@1(probing) e4
entity_name global_id 0 (none) caps allow *
2021-12-15T13:34:57.075+1000 7f6e1b417700 20 is_capable service=mon
command= read addr v2:172.16.21.103:3300/0 on cap allow *
2021-12-15T13:34:57.075+1000 7f6e1b417700 20 allow so far , doing grant
allow *
2021-12-15T13:34:57.075+1000 7f6e1b417700 20 allow all
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
handle_probe mon_probe(reply 8deaaacb-c581-4c10-b58c-0ab261aa2865 name
debian3 quorum 0,2 leader 0 paxos( fc 52724559 lc 52725302 ) mon_release
octopus) v7
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
handle_probe_reply mon.2 v2:172.16.21.103:3300/0 mon_probe(reply
8deaaacb-c581-4c10-b58c-0ab261aa2865 name debian3 quorum 0,2 leader 0
paxos( fc 52724559 lc 52725302 ) mon_release octopus) v7
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
monmap is e4: 3 mons at {debian1=[v2:
172.16.21.101:3300/0,v1:172.16.21.101:6789/0],debian2=[v2:
172.16.21.102:3300/0,v1:172.16.21.102:6789/0],debian3=[v2:
172.16.21.103:3300/0,v1:172.16.21.103:6789/0]}
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4 got
newer/committed monmap epoch 4, mine was 4
2021-12-15T13:34:57.075+1000 7f6e1b417700 10 mon.debian2@1(probing) e4
bootstrap
On the nautilus monitor I see
2021-12-15T13:57:03.866+1000 7f109cf23700 20 mon.debian1@0(leader) e4
_ms_dispatch existing session 0x55feee4f9b00 for mon.1
2021-12-15T13:57:03.866+1000 7f109cf23700 20 mon.debian1@0(leader) e4
entity_name global_id 0 (none) caps allow *
2021-12-15T13:57:03.866+1000 7f109cf23700 20 is_capable service=mon
command= read addr v2:172.16.21.102:3300/0 on cap allow *
2021-12-15T13:57:03.866+1000 7f109cf23700 20 allow so far , doing grant
allow *
2021-12-15T13:57:03.866+1000 7f109cf23700 20 allow all
2021-12-15T13:57:03.866+1000 7f109cf23700 10 mon.debian1@0(leader) e4
handle_probe mon_probe(probe 8deaaacb-c581-4c10-b58c-0ab261aa2865 name
debian2 new mon_release unknown) v8
2021-12-15T13:57:03.866+1000 7f109cf23700 10 mon.debian1@0(leader) e4
handle_probe_probe mon.1 v2:172.16.21.102:3300/0mon_probe(probe
8deaaacb-c581-4c10-b58c-0ab261aa2865 name debian2 new mon_release unknown)
v8 features 4540138292840890367
2021-12-15T13:57:03.866+1000 7f109cf23700 20 mon.debian1@0(leader) e4
_ms_dispatch existing session 0x55feee4f9b00 for mon.1
2021-12-15T13:57:03.866+1000 7f109cf23700 20 mon.debian1@0(leader) e4
entity_name global_id 0 (none) caps allow *
2021-12-15T13:57:03.866+1000 7f109cf23700 20 is_capable service=mon
command= read addr v2:172.16.21.102:3300/0 on cap allow *
2021-12-15T13:57:03.866+1000 7f109cf23700 20 allow so far , doing grant
allow *
2021-12-15T13:57:03.866+1000 7f109cf23700 20 allow all
2021-12-15T13:57:03.866+1000 7f109cf23700 10 mon.debian1@0(leader) e4
handle_probe mon_probe(probe 8deaaacb-c581-4c10-b58c-0ab261aa2865 name
debian2 new mon_release unknown) v8
2021-12-15T13:57:03.866+1000 7f109cf23700 10 mon.debian1@0(leader) e4
handle_probe_probe mon.1 v2:172.16.21.102:3300/0mon_probe(probe
8deaaacb-c581-4c10-b58c-0ab261aa2865 name debian2 new mon_release unknown)
v8 features 4540138292840890367
Pacific version 16.2.6 (test 16.2.7 with the same result), nautilus version
15.2.15.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
