.New Ceph cluster - cannot add additional monitor

Mike Carlson <mike@xxxxxxxxxxxx> · Tue, 9 Jun 2015 11:51:32 -0700

We have a new ceph cluster, and when I follow the guide (http://ceph.com/docs/master/start/quick-ceph-deploy/) during the section where you can add additional monitors, it fails, and it almost seems like its using an improper ip address

We have 4 nodes:
lts-mon
lts-osd1
lts-osd2
lts-osd3
Using, ceph-deploy, we have created a new cluster with lts-mon as the initial monitor:

ceph-deploy new lts-mon
ceph-deploy install lts-mon lts-osd1 lts-osd2 lts-osd3
ceph-deploy mon create-initial
ceph-deploy osd prepare ....
....
ceph-deploy mds lts-mon

The only modifications I made to ceph.conf were to include the public and cluster network settings, and set the osd pool default size:

[global]
fsid = 5ca0e0f5-d367-48b8-97b4-48e8b12fd517
mon_initial_members = lts-mon
mon_host = 10.5.68.236
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 3
public_network = 10.5.68.0/22
cluster_network = 10.1.1.0/24

This all seemed fine, and after adding in all of our osd's, ceph -s reports:
# ceph -s
    cluster f4adbd94-bf49-42f2-bd57-ebc7db9aa863
     health HEALTH_WARN
            too few PGs per OSD (1 < min 30)
     monmap e1: 1 mons at {lts-mon=10.5.68.236:6789/0}
            election epoch 1, quorum 0 lts-mon
     osdmap e471: 102 osds: 102 up, 102 in
      pgmap v973: 64 pgs, 1 pools, 0 bytes data, 0 objects
            515 GB used, 370 TB / 370 TB avail
                  64 active+clean

We have not defined the default pg so the warning seems okay for now

The problem we have is when adding a new monitor:
ceph-deploy mon create lts-osd1

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /usr/local/bin/ceph-deploy mon create lts-osd1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts lts-osd1
[ceph_deploy.mon][DEBUG ] detecting platform for host lts-osd1 ...
[lts-osd1][DEBUG ] connection detected need for sudo
[lts-osd1][DEBUG ] connected to host: lts-osd1 
[lts-osd1][DEBUG ] detect platform information from remote host
[lts-osd1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: Ubuntu 14.04 trusty
[lts-osd1][DEBUG ] determining if provided host has same hostname in remote
[lts-osd1][DEBUG ] get remote short hostname
[lts-osd1][DEBUG ] deploying mon to lts-osd1
[lts-osd1][DEBUG ] get remote short hostname
[lts-osd1][DEBUG ] remote hostname: lts-osd1
[lts-osd1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[lts-osd1][DEBUG ] create the mon path if it does not exist
[lts-osd1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-lts-osd1/done
[lts-osd1][DEBUG ] create a done file to avoid re-doing the mon deployment
[lts-osd1][DEBUG ] create the init path if it does not exist
[lts-osd1][DEBUG ] locating the `service` executable...
[lts-osd1][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=lts-osd1
[lts-osd1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.lts-osd1.asok mon_status
[lts-osd1][DEBUG ] ********************************************************************************
[lts-osd1][DEBUG ] status for monitor: mon.lts-osd1
[lts-osd1][DEBUG ] {
[lts-osd1][DEBUG ]   "election_epoch": 0, 
[lts-osd1][DEBUG ]   "extra_probe_peers": [
[lts-osd1][DEBUG ]     "10.5.68.236:6789/0"
[lts-osd1][DEBUG ]   ], 
[lts-osd1][DEBUG ]   "monmap": {
[lts-osd1][DEBUG ]     "created": "0.000000", 
[lts-osd1][DEBUG ]     "epoch": 0, 
[lts-osd1][DEBUG ]     "fsid": "5ca0e0f5-d367-48b8-97b4-48e8b12fd517", 
[lts-osd1][DEBUG ]     "modified": "0.000000", 
[lts-osd1][DEBUG ]     "mons": [
[lts-osd1][DEBUG ]       {
[lts-osd1][DEBUG ]         "addr": "0.0.0.0:0/1", 
[lts-osd1][DEBUG ]         "name": "lts-mon", 
[lts-osd1][DEBUG ]         "rank": 0
[lts-osd1][DEBUG ]       }
[lts-osd1][DEBUG ]     ]
[lts-osd1][DEBUG ]   }, 
[lts-osd1][DEBUG ]   "name": "lts-osd1", 
[lts-osd1][DEBUG ]   "outside_quorum": [], 
[lts-osd1][DEBUG ]   "quorum": [], 
[lts-osd1][DEBUG ]   "rank": -1, 
[lts-osd1][DEBUG ]   "state": "probing", 
[lts-osd1][DEBUG ]   "sync_provider": []
[lts-osd1][DEBUG ] }
[lts-osd1][DEBUG ] ********************************************************************************
[lts-osd1][INFO  ] monitor: mon.lts-osd1 is currently at the state of probing
[lts-osd1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.lts-osd1.asok mon_status
[lts-osd1][WARNIN] lts-osd1 is not defined in `mon initial members`
[lts-osd1][WARNIN] monitor lts-osd1 does not exist in monmap

the monitor I was trying to add shows:
2015-06-09 11:33:24.661466 7fef2a806700  0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption
2015-06-09 11:33:24.661478 7fef2a806700  0 -- 10.5.68.229:6789/0 >> 10.5.68.236:6789/0 pipe(0x3571000 sd=13 :40912 s=1 pgs=0 cs=0 l=0 c=0x34083c0).failed verifying authorize reply
2015-06-09 11:33:24.763579 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2015-06-09 11:33:24.763651 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2015-06-09 11:33:25.825163 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2015-06-09 11:33:25.825259 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2015-06-09 11:33:26.661737 7fef2a806700  0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption
2015-06-09 11:33:26.661750 7fef2a806700  0 -- 10.5.68.229:6789/0 >> 10.5.68.236:6789/0 pipe(0x3571000 sd=13 :40914 s=1 pgs=0 cs=0 l=0 c=0x34083c0).failed verifying authorize reply
2015-06-09 11:33:26.887973 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2015-06-09 11:33:26.888047 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2015-06-09 11:33:27.950014 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2015-06-09 11:33:27.950113 7fef2eb83700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished

All of our google searching seems to indicate that there may be a clock skew, but, the clocks are matched within .001 seconds

Any assistance is much appreciated, thanks,

Mike C
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com