|
Hi folks- I’m trying to deploy 0.94.2 (Hammer) onto CentOS7. I used to be pretty good at this on Ubuntu but it has been a while. Anyway, my monitors are not forming quorum, and I’m not sure why.
They can definitely all ping each other and such. Any thoughts on specific problems in the output below, or just general causes for monitors not forming quorum, or where to get more debug information on what is going wrong? Thanks!! [root@bdca151 ceph]# ceph-deploy mon create-initial bdca15{0,2,3} [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.25): /bin/ceph-deploy mon create-initial bdca150 bdca152 bdca153 [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts bdca150 bdca152 bdca153 [ceph_deploy.mon][DEBUG ] detecting platform for host bdca150 ... [bdca150][DEBUG ] connected to host: bdca150 [bdca150][DEBUG ] detect platform information from remote host [bdca150][DEBUG ] detect machine type [ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.1.1503 Core [bdca150][DEBUG ] determining if provided host has same hostname in remote [bdca150][DEBUG ] get remote short hostname [bdca150][DEBUG ] deploying mon to bdca150 [bdca150][DEBUG ] get remote short hostname [bdca150][DEBUG ] remote hostname: bdca150 [bdca150][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [bdca150][DEBUG ] create the mon path if it does not exist [bdca150][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-bdca150/done [bdca150][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-bdca150/done [bdca150][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-bdca150.mon.keyring [bdca150][DEBUG ] create the monitor keyring file [bdca150][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i bdca150 --keyring /var/lib/ceph/tmp/ceph-bdca150.mon.keyring [bdca150][DEBUG ] ceph-mon: renaming mon.noname-a 10.1.0.150:6789/0 to mon.bdca150 [bdca150][DEBUG ] ceph-mon: set fsid to 770514ba-65e6-475b-8d43-ad6ee850ead6 [bdca150][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-bdca150 for mon.bdca150 [bdca150][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-bdca150.mon.keyring [bdca150][DEBUG ] create a done file to avoid re-doing the mon deployment [bdca150][DEBUG ] create the init path if it does not exist [bdca150][DEBUG ] locating the `service` executable... [bdca150][INFO ] Running command: /usr/sbin/service ceph -c /etc/ceph/ceph.conf start mon.bdca150 [bdca150][DEBUG ] === mon.bdca150 === [bdca150][DEBUG ] Starting Ceph mon.bdca150 on bdca150... [bdca150][WARNIN] Running as unit run-52328.service. [bdca150][DEBUG ] Starting ceph-create-keys on bdca150... [bdca150][INFO ] Running command: systemctl enable ceph [bdca150][WARNIN] ceph.service is not a native service, redirecting to /sbin/chkconfig. [bdca150][WARNIN] Executing /sbin/chkconfig ceph on [bdca150][WARNIN] The unit files have no [Install] section. They are not meant to be enabled [bdca150][WARNIN] using systemctl. [bdca150][WARNIN] Possible reasons for having this kind of units are: [bdca150][WARNIN] 1) A unit may be statically enabled by being symlinked from another unit's [bdca150][WARNIN] .wants/ or .requires/ directory. [bdca150][WARNIN] 2) A unit's purpose may be to act as a helper for some other unit which has [bdca150][WARNIN] a requirement dependency on it. [bdca150][WARNIN] 3) A unit may be started when needed via activation (socket, path, timer, [bdca150][WARNIN] D-Bus, udev, scripted systemctl call, ...). [bdca150][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca150.asok mon_status [bdca150][DEBUG ] ******************************************************************************** [bdca150][DEBUG ] status for monitor: mon.bdca150 [bdca150][DEBUG ] { [bdca150][DEBUG ] "election_epoch": 0, [bdca150][DEBUG ] "extra_probe_peers": [ [bdca150][DEBUG ] "10.1.0.152:6789/0", [bdca150][DEBUG ] "10.1.0.153:6789/0" [bdca150][DEBUG ] ], [bdca150][DEBUG ] "monmap": { [bdca150][DEBUG ] "created": "0.000000", [bdca150][DEBUG ] "epoch": 0, [bdca150][DEBUG ] "fsid": "770514ba-65e6-475b-8d43-ad6ee850ead6", [bdca150][DEBUG ] "modified": "0.000000", [bdca150][DEBUG ] "mons": [ [bdca150][DEBUG ] { [bdca150][DEBUG ] "addr": "10.1.0.150:6789/0", [bdca150][DEBUG ] "name": "bdca150", [bdca150][DEBUG ] "rank": 0 [bdca150][DEBUG ] }, [bdca150][DEBUG ] { [bdca150][DEBUG ] "addr": "0.0.0.0:0/1", [bdca150][DEBUG ] "name": "bdca152", [bdca150][DEBUG ] "rank": 1 [bdca150][DEBUG ] }, [bdca150][DEBUG ] { [bdca150][DEBUG ] "addr": "0.0.0.0:0/2", [bdca150][DEBUG ] "name": "bdca153", [bdca150][DEBUG ] "rank": 2 [bdca150][DEBUG ] } [bdca150][DEBUG ] ] [bdca150][DEBUG ] }, [bdca150][DEBUG ] "name": "bdca150", [bdca150][DEBUG ] "outside_quorum": [ [bdca150][DEBUG ] "bdca150" [bdca150][DEBUG ] ], [bdca150][DEBUG ] "quorum": [], [bdca150][DEBUG ] "rank": 0, [bdca150][DEBUG ] "state": "probing", [bdca150][DEBUG ] "sync_provider": [] [bdca150][DEBUG ] } [bdca150][DEBUG ] ******************************************************************************** [bdca150][INFO ] monitor: mon.bdca150 is running [bdca150][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca150.asok mon_status [ceph_deploy.mon][DEBUG ] detecting platform for host bdca152 ... [bdca152][DEBUG ] connected to host: bdca152 [bdca152][DEBUG ] detect platform information from remote host [bdca152][DEBUG ] detect machine type [ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.1.1503 Core [bdca152][DEBUG ] determining if provided host has same hostname in remote [bdca152][DEBUG ] get remote short hostname [bdca152][DEBUG ] deploying mon to bdca152 [bdca152][DEBUG ] get remote short hostname [bdca152][DEBUG ] remote hostname: bdca152 [bdca152][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [bdca152][DEBUG ] create the mon path if it does not exist [bdca152][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-bdca152/done [bdca152][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-bdca152/done [bdca152][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-bdca152.mon.keyring [bdca152][DEBUG ] create the monitor keyring file [bdca152][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i bdca152 --keyring /var/lib/ceph/tmp/ceph-bdca152.mon.keyring [bdca152][DEBUG ] ceph-mon: renaming mon.noname-b 10.1.0.152:6789/0 to mon.bdca152 [bdca152][DEBUG ] ceph-mon: set fsid to 770514ba-65e6-475b-8d43-ad6ee850ead6 [bdca152][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-bdca152 for mon.bdca152 [bdca152][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-bdca152.mon.keyring [bdca152][DEBUG ] create a done file to avoid re-doing the mon deployment [bdca152][DEBUG ] create the init path if it does not exist [bdca152][DEBUG ] locating the `service` executable... [bdca152][INFO ] Running command: /usr/sbin/service ceph -c /etc/ceph/ceph.conf start mon.bdca152 [bdca152][DEBUG ] === mon.bdca152 === [bdca152][DEBUG ] Starting Ceph mon.bdca152 on bdca152... [bdca152][WARNIN] Running as unit run-49587.service. [bdca152][DEBUG ] Starting ceph-create-keys on bdca152... [bdca152][INFO ] Running command: systemctl enable ceph [bdca152][WARNIN] ceph.service is not a native service, redirecting to /sbin/chkconfig. [bdca152][WARNIN] Executing /sbin/chkconfig ceph on [bdca152][WARNIN] The unit files have no [Install] section. They are not meant to be enabled [bdca152][WARNIN] using systemctl. [bdca152][WARNIN] Possible reasons for having this kind of units are: [bdca152][WARNIN] 1) A unit may be statically enabled by being symlinked from another unit's [bdca152][WARNIN] .wants/ or .requires/ directory. [bdca152][WARNIN] 2) A unit's purpose may be to act as a helper for some other unit which has [bdca152][WARNIN] a requirement dependency on it. [bdca152][WARNIN] 3) A unit may be started when needed via activation (socket, path, timer, [bdca152][WARNIN] D-Bus, udev, scripted systemctl call, ...). [bdca152][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca152.asok mon_status [bdca152][DEBUG ] ******************************************************************************** [bdca152][DEBUG ] status for monitor: mon.bdca152 [bdca152][DEBUG ] { [bdca152][DEBUG ] "election_epoch": 0, [bdca152][DEBUG ] "extra_probe_peers": [ [bdca152][DEBUG ] "10.1.0.150:6789/0", [bdca152][DEBUG ] "10.1.0.153:6789/0" [bdca152][DEBUG ] ], [bdca152][DEBUG ] "monmap": { [bdca152][DEBUG ] "created": "0.000000", [bdca152][DEBUG ] "epoch": 0, [bdca152][DEBUG ] "fsid": "770514ba-65e6-475b-8d43-ad6ee850ead6", [bdca152][DEBUG ] "modified": "0.000000", [bdca152][DEBUG ] "mons": [ [bdca152][DEBUG ] { [bdca152][DEBUG ] "addr": "10.1.0.152:6789/0", [bdca152][DEBUG ] "name": "bdca152", [bdca152][DEBUG ] "rank": 0 [bdca152][DEBUG ] }, [bdca152][DEBUG ] { [bdca152][DEBUG ] "addr": "0.0.0.0:0/1", [bdca152][DEBUG ] "name": "bdca150", [bdca152][DEBUG ] "rank": 1 [bdca152][DEBUG ] }, [bdca152][DEBUG ] { [bdca152][DEBUG ] "addr": "0.0.0.0:0/2", [bdca152][DEBUG ] "name": "bdca153", [bdca152][DEBUG ] "rank": 2 [bdca152][DEBUG ] } [bdca152][DEBUG ] ] [bdca152][DEBUG ] }, [bdca152][DEBUG ] "name": "bdca152", [bdca152][DEBUG ] "outside_quorum": [ [bdca152][DEBUG ] "bdca152" [bdca152][DEBUG ] ], [bdca152][DEBUG ] "quorum": [], [bdca152][DEBUG ] "rank": 0, [bdca152][DEBUG ] "state": "probing", [bdca152][DEBUG ] "sync_provider": [] [bdca152][DEBUG ] } [bdca152][DEBUG ] ******************************************************************************** [bdca152][INFO ] monitor: mon.bdca152 is running [bdca152][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca152.asok mon_status [ceph_deploy.mon][DEBUG ] detecting platform for host bdca153 ... [bdca153][DEBUG ] connected to host: bdca153 [bdca153][DEBUG ] detect platform information from remote host [bdca153][DEBUG ] detect machine type [ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.1.1503 Core [bdca153][DEBUG ] determining if provided host has same hostname in remote [bdca153][DEBUG ] get remote short hostname [bdca153][DEBUG ] deploying mon to bdca153 [bdca153][DEBUG ] get remote short hostname [bdca153][DEBUG ] remote hostname: bdca153 [bdca153][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [bdca153][DEBUG ] create the mon path if it does not exist [bdca153][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-bdca153/done [bdca153][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-bdca153/done [bdca153][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-bdca153.mon.keyring [bdca153][DEBUG ] create the monitor keyring file [bdca153][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i bdca153 --keyring /var/lib/ceph/tmp/ceph-bdca153.mon.keyring [bdca153][DEBUG ] ceph-mon: renaming mon.noname-c 10.1.0.153:6789/0 to mon.bdca153 [bdca153][DEBUG ] ceph-mon: set fsid to 770514ba-65e6-475b-8d43-ad6ee850ead6 [bdca153][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-bdca153 for mon.bdca153 [bdca153][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-bdca153.mon.keyring [bdca153][DEBUG ] create a done file to avoid re-doing the mon deployment [bdca153][DEBUG ] create the init path if it does not exist [bdca153][DEBUG ] locating the `service` executable... [bdca153][INFO ] Running command: /usr/sbin/service ceph -c /etc/ceph/ceph.conf start mon.bdca153 [bdca153][DEBUG ] === mon.bdca153 === [bdca153][DEBUG ] Starting Ceph mon.bdca153 on bdca153... [bdca153][WARNIN] Running as unit run-49206.service. [bdca153][DEBUG ] Starting ceph-create-keys on bdca153... [bdca153][INFO ] Running command: systemctl enable ceph [bdca153][WARNIN] ceph.service is not a native service, redirecting to /sbin/chkconfig. [bdca153][WARNIN] Executing /sbin/chkconfig ceph on [bdca153][WARNIN] The unit files have no [Install] section. They are not meant to be enabled [bdca153][WARNIN] using systemctl. [bdca153][WARNIN] Possible reasons for having this kind of units are: [bdca153][WARNIN] 1) A unit may be statically enabled by being symlinked from another unit's [bdca153][WARNIN] .wants/ or .requires/ directory. [bdca153][WARNIN] 2) A unit's purpose may be to act as a helper for some other unit which has [bdca153][WARNIN] a requirement dependency on it. [bdca153][WARNIN] 3) A unit may be started when needed via activation (socket, path, timer, [bdca153][WARNIN] D-Bus, udev, scripted systemctl call, ...). [bdca153][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca153.asok mon_status [bdca153][DEBUG ] ******************************************************************************** [bdca153][DEBUG ] status for monitor: mon.bdca153 [bdca153][DEBUG ] { [bdca153][DEBUG ] "election_epoch": 0, [bdca153][DEBUG ] "extra_probe_peers": [ [bdca153][DEBUG ] "10.1.0.150:6789/0", [bdca153][DEBUG ] "10.1.0.152:6789/0" [bdca153][DEBUG ] ], [bdca153][DEBUG ] "monmap": { [bdca153][DEBUG ] "created": "0.000000", [bdca153][DEBUG ] "epoch": 0, [bdca153][DEBUG ] "fsid": "770514ba-65e6-475b-8d43-ad6ee850ead6", [bdca153][DEBUG ] "modified": "0.000000", [bdca153][DEBUG ] "mons": [ [bdca153][DEBUG ] { [bdca153][DEBUG ] "addr": "10.1.0.153:6789/0", [bdca153][DEBUG ] "name": "bdca153", [bdca153][DEBUG ] "rank": 0 [bdca153][DEBUG ] }, [bdca153][DEBUG ] { [bdca153][DEBUG ] "addr": "0.0.0.0:0/1", [bdca153][DEBUG ] "name": "bdca150", [bdca153][DEBUG ] "rank": 1 [bdca153][DEBUG ] }, [bdca153][DEBUG ] { [bdca153][DEBUG ] "addr": "0.0.0.0:0/2", [bdca153][DEBUG ] "name": "bdca152", [bdca153][DEBUG ] "rank": 2 [bdca153][DEBUG ] } [bdca153][DEBUG ] ] [bdca153][DEBUG ] }, [bdca153][DEBUG ] "name": "bdca153", [bdca153][DEBUG ] "outside_quorum": [ [bdca153][DEBUG ] "bdca153" [bdca153][DEBUG ] ], [bdca153][DEBUG ] "quorum": [], [bdca153][DEBUG ] "rank": 0, [bdca153][DEBUG ] "state": "probing", [bdca153][DEBUG ] "sync_provider": [] [bdca153][DEBUG ] } [bdca153][DEBUG ] ******************************************************************************** [bdca153][INFO ] monitor: mon.bdca153 is running [bdca153][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca153.asok mon_status [ceph_deploy.mon][INFO ] processing monitor mon.bdca150 [bdca150][DEBUG ] connected to host: bdca150 [bdca150][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca150.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca150 monitor is not yet in quorum, tries left: 5 [ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying [bdca150][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca150.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca150 monitor is not yet in quorum, tries left: 4 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [bdca150][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca150.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca150 monitor is not yet in quorum, tries left: 3 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [bdca150][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca150.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca150 monitor is not yet in quorum, tries left: 2 [ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying [bdca150][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca150.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca150 monitor is not yet in quorum, tries left: 1 [ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying [ceph_deploy.mon][INFO ] processing monitor mon.bdca152 [bdca152][DEBUG ] connected to host: bdca152 [bdca152][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca152.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca152 monitor is not yet in quorum, tries left: 5 [ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying [bdca152][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca152.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca152 monitor is not yet in quorum, tries left: 4 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [bdca152][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca152.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca152 monitor is not yet in quorum, tries left: 3 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [bdca152][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca152.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca152 monitor is not yet in quorum, tries left: 2 [ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying [bdca152][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca152.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca152 monitor is not yet in quorum, tries left: 1 [ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying [ceph_deploy.mon][INFO ] processing monitor mon.bdca153 [bdca153][DEBUG ] connected to host: bdca153 [bdca153][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca153.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca153 monitor is not yet in quorum, tries left: 5 [ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying [bdca153][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca153.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca153 monitor is not yet in quorum, tries left: 4 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [bdca153][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca153.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca153 monitor is not yet in quorum, tries left: 3 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [bdca153][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca153.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca153 monitor is not yet in quorum, tries left: 2 [ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying [bdca153][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.bdca153.asok mon_status [ceph_deploy.mon][WARNIN] mon.bdca153 monitor is not yet in quorum, tries left: 1 [ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying [ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum: [ceph_deploy.mon][ERROR ] bdca153 [ceph_deploy.mon][ERROR ] bdca152 [ceph_deploy.mon][ERROR ] bdca150 [root@bdca151 ceph]# |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
