Monitors not reaching quorum. (SELinux off, IPtables off, can see tcp traffic)

[Cameron.Scrace@xxxxxxxxxxxx]

I am trying to deploy a new ceph cluster
and my monitors are not reaching quorum. SELinux is off, firewalls are
off, I can see traffic between the nodes on port 6789 but when I use the
admin socket to force a re-election only the monitor I send the request
to shows the new election in its logs. My logs are filled entirely of the
following two lines:

2015-06-02 11:31:56.447975 7f795b17a700
 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin
socket' cmd='mon_status' args=[]: dispatch
2015-06-02 11:31:56.448272 7f795b17a700
 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin
socket' cmd=mon_status args=[]: finished

Querying the admin socket with mon_status
(the other two are the similar but with their hostnames and rank):

{
    "name": "wcm1",
    "rank": 0,
    "state": "probing",
    "election_epoch":
1,
    "quorum": [],
    "outside_quorum":
[
        "wcm1"
    ],
    "extra_probe_peers":
[],
    "sync_provider":
[],
    "monmap": {
        "epoch":
0,
        "fsid":
"adb8c500-122e-49fd-9c1e-a99af7832307",
        "modified":
"2015-06-02 10:43:41.467811",
        "created":
"2015-06-02 10:43:41.467811",
        "mons":
[
           
{
           
    "rank": 0,
           
    "name": "wcm1",
           
    "addr": "10.1.226.64:6789\/0"
           
},
           
{
           
    "rank": 1,
           
    "name": "wcm2",
           
    "addr": "10.1.226.65:6789\/0"
           
},
           
{
           
    "rank": 2,
           
    "name": "wcm3",
           
    "addr": "10.1.226.66:6789\/0"
           
}
        ]
    }
}

Any suggestions on what could be the
issue?

Regards,

Cameron Scrace
Infrastructure Engineer

Mobile +64 22 610 4629
Phone  +64 4 462 5085 
Email  cameron.scrace@xxxxxxxxxxxx
Solnet Solutions Limited
Level 12, Solnet House
70 The Terrace, Wellington 6011
PO Box 397, Wellington 6140

www.solnet.co.nzAttention:
This email may contain information intended for the sole use of
the original recipient. Please respect this when sharing or
disclosing this email's contents with any third party. If you
believe you have received this email in error, please delete it
and notify the sender or postmaster@xxxxxxxxxxxxxxxxxxxxx as
soon as possible. The content of this email does not necessarily
reflect the views of Solnet Solutions Ltd.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com