On 06/02/2015 01:42 AM, Cameron.Scrace@xxxxxxxxxxxx wrote:
> I am trying to deploy a new ceph cluster and my monitors are not
> reaching quorum. SELinux is off, firewalls are off, I can see traffic
> between the nodes on port 6789 but when I use the admin socket to force
> a re-election only the monitor I send the request to shows the new
> election in its logs. My logs are filled entirely of the following two
> lines:
>
> 2015-06-02 11:31:56.447975 7f795b17a700 0 log_channel(audit) log [DBG]
> : from='admin socket' entity='admin socket' cmd='mon_status' args=[]:
> dispatch
> 2015-06-02 11:31:56.448272 7f795b17a700 0 log_channel(audit) log [DBG]
> : from='admin socket' entity='admin socket' cmd=mon_status args=[]:
> finished
You are running on default debug levels, so you'll hardly get anything
more than that. I suggest setting 'debug mon = 10' and 'debug ms = 1'
for added verbosity and come back to us with the logs.
There are many reasons for this, but the more common are due to the
monitors not being able to communicate with each other. Given you see
traffic between the monitors, I'm inclined to assume that the other two
monitors do not have each other on the monmap or, if they do know each
other, either 1) the monitor's auth keys do not match, or 2) the probe
timeout is being triggered before they successfully manage to find
enough monitors to trigger an election -- which may be due to latency.
Logs will tells us more.
-Joao
> Querying the admin socket with mon_status (the other two are the similar
> but with their hostnames and rank):
>
> {
> "name": "wcm1",
> "rank": 0,
> "state": "probing",
> "election_epoch": 1,
> "quorum": [],
> "outside_quorum": [
> "wcm1"
> ],
> "extra_probe_peers": [],
> "sync_provider": [],
> "monmap": {
> "epoch": 0,
> "fsid": "adb8c500-122e-49fd-9c1e-a99af7832307",
> "modified": "2015-06-02 10:43:41.467811",
> "created": "2015-06-02 10:43:41.467811",
> "mons": [
> {
> "rank": 0,
> "name": "wcm1",
> "addr": "10.1.226.64:6789\/0"
> },
> {
> "rank": 1,
> "name": "wcm2",
> "addr": "10.1.226.65:6789\/0"
> },
> {
> "rank": 2,
> "name": "wcm3",
> "addr": "10.1.226.66:6789\/0"
> }
> ]
> }
> }
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
