In our lab-setup, I'm simulating the future migration of centos 7 + ceph
14.2.x to cepntos 8 + ceph 15.2.x.
At the moment, I upgraded one of the nodes, which is a combined
mon+mgr+mds+osd, to el8 + 15.2.2. The other node (also a combined one) is
still on el7 + 14.2.9.
The osd was detected and re-added easily, as were the mgr and mds. The mon
won't add.
I use the following (manual) method:
1. ceph auth get mon. -o /tmp/kr (exports correctly)
2. ceph mon getmap -o /tmp/mm (exports correctly)
3. sudo ceph-mon -i ontw-ceph01 --mkfs --monmap /tmp/mm --keyring
/tmp/kr (creates file system + rocksdb correctly in
"/var/lib/ceph/mon/ceph-ontw-ceph01")
4. systemctl start ceph-mon@ontw-ceph01.service
ceph-mon starts up successfully, but won't enter the cluster.
In the other (14.2.9) ceph-mon log, I see loads of this message (multiple
per :
--------------
2020-05-25 13:07:38.075 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3
adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of
hints
2020-05-25 13:07:38.075 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3
adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of
hints
2020-05-25 13:07:38.076 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3
adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of
hints
2020-05-25 13:07:38.076 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3
adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of
hints
2020-05-25 13:07:38.077 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3
adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of
hints
etc...
--------------
The 3300 and 6789 ports are open and reacheable on both nodes and they can
reach and connect to each other.
192.168.100.60 = ontw-ceph01, upgraded 15.2.2-node
192.168.100.61 = ontw-ceph02, active 14.2.9-node
When I start ceph-mon in debug mode @ ceph01, I see the following happening:
--------------
2020-05-25T13:07:35.031+0200 7f712e1d46c0 0 starting mon.ontw-ceph01 rank
-1 at public addrs [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] at
bind addrs [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] mon_data
/var/lib/ceph/mon/ceph-ontw-ceph01 fsid f3c3f099-2940-4074-a7fe-1aea6259f67b
2020-05-25T13:07:35.033+0200 7f712e1d46c0 1 mon.ontw-ceph01@-1(???) e1
preinit fsid f3c3f099-2940-4074-a7fe-1aea6259f67b
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
check_fsid cluster_uuid contains 'f3c3f099-2940-4074-a7fe-1aea6259f67b'
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
features compat={},rocompat={},incompat={1=initial feature set
(~v.18),3=single paxos with k/v store (v0.?)}
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
calc_quorum_requirements required_features 2449958197560098820
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
required_features 2449958197560098820
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
has_ever_joined = 0
2020-05-25T13:07:35.033+0200 7f712e1d46c0 1 mon.ontw-ceph01@-1(???) e1
initial_members ontw-ceph02,ontw-ceph01, filtering seed monmap
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
monmap is e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2:
192.168.100.61:3300/0,v1:192.168.100.61:6789/0]}
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
extra probe peers
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
sync_last_committed_floor 0
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
init_paxos
2020-05-25T13:07:35.033+0200 7f712e1d46c0 5 mon.ontw-ceph01@-1(???).mds e0
Unable to load 'last_metadata'
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).health
init
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).config
init
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
refresh_from_paxos
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
refresh_from_paxos no cluster_fingerprint
2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).log v0
update_from_paxos
2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).log v0
update_from_paxos version 0 summary v 0
2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).auth
v0 update_from_paxos
2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).config
load_config got 0 keys
2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).mgrstat
0
2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).mgrstat
check_subs
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).health
update_from_paxos
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1
loading initial keyring to bootstrap authentication for mkfs
2020-05-25T13:07:35.035+0200 7f712e1d46c0 2 mon.ontw-ceph01@-1(???) e1 init
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???).mgr e0
prime_mgr_client
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
bootstrap
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
sync_reset_requester
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
unregister_cluster_logger - not registered
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
monmap e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2:
192.168.100.61:3300/0,v1:192.168.100.61:6789/0]}
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
_reset
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing).auth
v0 _set_mon_num_rank num 0 rank 0
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
timecheck_finish
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
scrub_event_cancel
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
scrub_reset
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
reset_probe_timeout 0x563148b1eb60 after 2 seconds
2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1
probing other monitors
2020-05-25T13:07:35.035+0200 7f712e1d46c0 0 -- [v2:
192.168.100.60:3300/0,v1:192.168.100.60:6789/0] send_to message
mon_probe(probe f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph01 new
mon_release octopus) v7 with empty dest
2020-05-25T13:07:35.036+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1
get_authorizer for mon
2020-05-25T13:07:35.238+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1
get_authorizer for mon
2020-05-25T13:07:35.640+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1
get_authorizer for mon
2020-05-25T13:07:36.442+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1
get_authorizer for mon
2020-05-25T13:07:37.035+0200 7f711a308700 4 mon.ontw-ceph01@-1(probing) e1
probe_timeout 0x563148b1eb60
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
bootstrap
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
sync_reset_requester
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
unregister_cluster_logger - not registered
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
monmap e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2:
192.168.100.61:3300/0,v1:192.168.100.61:6789/0]}
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
_reset
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing).auth
v0 _set_mon_num_rank num 0 rank 0
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
timecheck_finish
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
scrub_event_cancel
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
scrub_reset
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
reset_probe_timeout 0x563148b1f7c0 after 2 seconds
2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1
probing other monitors
2020-05-25T13:07:37.035+0200 7f711a308700 0 -- [v2:
192.168.100.60:3300/0,v1:192.168.100.60:6789/0] send_to message
mon_probe(probe f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph01 new
mon_release octopus) v7 with empty dest
2020-05-25T13:07:38.045+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1
get_authorizer for mon
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1
_ms_dispatch new session 0x563149893680 MonSession(mon.0 [v2:
192.168.100.61:3300/0,v1:192.168.100.61:6789/0] is open , features
0x3f01cfb8ffacffff (luminous)) features 0x3f01cfb8ffacffff
2020-05-25T13:07:38.047+0200 7f7117b03700 5 mon.ontw-ceph01@-1(probing) e1
_ms_dispatch setting monitor caps on this connection
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1
handle_probe mon_probe(reply f3c3f099-2940-4074-a7fe-1aea6259f67b name
ontw-ceph02 quorum 0 paxos( fc 304966 lc 305658 ) mon_release nautilus) v7
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1
handle_probe_reply mon.0 v2:192.168.100.61:3300/0 mon_probe(reply
f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph02 quorum 0 paxos( fc
304966 lc 305658 ) mon_release nautilus) v7
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1
monmap is e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2:
192.168.100.61:3300/0,v1:192.168.100.61:6789/0]}
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1
got newer/committed monmap epoch 3, mine was 1
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
bootstrap
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
sync_reset_requester
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
unregister_cluster_logger - not registered
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
cancel_probe_timeout 0x563148b1f7c0
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
monmap e3: 1 mons at {ontw-ceph02=[v2:
192.168.100.61:3300/0,v1:192.168.100.61:6789/0]}
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
_reset
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing).auth
v0 _set_mon_num_rank num 0 rank 0
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
timecheck_finish
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
scrub_event_cancel
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
scrub_reset
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
cancel_probe_timeout (none scheduled)
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
reset_probe_timeout 0x563148b1f7c0 after 2 seconds
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
probing other monitors
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
handle_probe mon_probe(reply f3c3f099-2940-4074-a7fe-1aea6259f67b name
ontw-ceph02 quorum 0 paxos( fc 304966 lc 305658 ) mon_release nautilus) v7
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
handle_probe_reply mon.0 v2:192.168.100.61:3300/0 mon_probe(reply
f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph02 quorum 0 paxos( fc
304966 lc 305658 ) mon_release nautilus) v7
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
monmap is e3: 1 mons at {ontw-ceph02=[v2:
192.168.100.61:3300/0,v1:192.168.100.61:6789/0]}
2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3
got newer/committed monmap epoch 3, mine was 3
--------------
I'm a bit lost at what's (not) happening. Hope you can help!
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
