Re: My misadventure in trying to install Ceph Emperor

[Ron Gage <ron@xxxxxxxxxxx>]

 

[ceph@cm my-cluster]$ ceph-deploy mon create cm
[ceph_deploy.cli][INFO  ] Invoked (1.3.5): /usr/bin/ceph-deploy mon create cm
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cm
[ceph_deploy.mon][DEBUG ] detecting platform for host cm ...
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: CentOS 6.5 Final
[cm][DEBUG ] determining if provided host has same hostname in remote
[cm][DEBUG ] get remote short hostname
[cm][DEBUG ] deploying mon to cm
[cm][DEBUG ] get remote short hostname
[cm][DEBUG ] remote hostname: cm
[cm][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cm][DEBUG ] create the mon path if it does not exist
[cm][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cm/done
[cm][DEBUG ] create a done file to avoid re-doing the mon deployment
[cm][DEBUG ] create the init path if it does not exist
[cm][DEBUG ] locating the `service` executable...
[cm][INFO  ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cm
[cm][DEBUG ] === mon.cm ===
[cm][DEBUG ] Starting Ceph mon.cm on cm...
[cm][DEBUG ] Starting ceph-create-keys on cm...
[cm][WARNIN] No data was received after 7 seconds, disconnecting...
[cm][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status
[cm][DEBUG ] ********************************************************************************
[cm][DEBUG ] status for monitor: mon.cm
[cm][DEBUG ] {
[cm][DEBUG ]   "election_epoch": 1,
[cm][DEBUG ]   "extra_probe_peers": [],
[cm][DEBUG ]   "monmap": {
[cm][DEBUG ]     "created": "0.000000",
[cm][DEBUG ]     "epoch": 1,
[cm][DEBUG ]     "fsid": "b11875f0-e611-40b9-ae81-e7cba7586ed5",
[cm][DEBUG ]     "modified": "0.000000",
[cm][DEBUG ]     "mons": [
[cm][DEBUG ]       {
[cm][DEBUG ]         "addr": "172.24.12.91:6789/0",
[cm][DEBUG ]         "name": "cm",
[cm][DEBUG ]         "rank": 0
[cm][DEBUG ]       }
[cm][DEBUG ]     ]
[cm][DEBUG ]   },
[cm][DEBUG ]   "name": "cm",
[cm][DEBUG ]   "outside_quorum": [],
[cm][DEBUG ]   "quorum": [
[cm][DEBUG ]     0
[cm][DEBUG ]   ],
[cm][DEBUG ]   "rank": 0,
[cm][DEBUG ]   "state": "leader",
[cm][DEBUG ]   "sync_provider": []
[cm][DEBUG ] }
[cm][DEBUG ] ********************************************************************************
[cm][INFO  ] monitor: mon.cm is running
[cm][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status
[ceph@cm my-cluster]$ ^C

 

[ceph@cm my-cluster]$ ceph-deploy gatherkeys cm
[ceph_deploy.cli][INFO  ] Invoked (1.3.5): /usr/bin/ceph-deploy gatherkeys cm
[ceph_deploy.gatherkeys][DEBUG ] Checking cm for /etc/ceph/ceph.client.admin.keyring
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[cm][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['cm']
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking cm for /var/lib/ceph/bootstrap-osd/ceph.keyring
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[cm][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm']
[ceph_deploy.gatherkeys][DEBUG ] Checking cm for /var/lib/ceph/bootstrap-mds/ceph.keyring
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[cm][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm']
[ceph@cm my-cluster]$

 

 

 

On February 14, 2014 at 8:20 AM Ashish Chandra <mail.ashishchandra@xxxxxxxxx> wrote:

Hi Ron,

Can you try running :

ceph-deploy mon create {ceph-node}

ceph-deploy gatherkeys {ceph-node}

as two different commands, instead of running

ceph-deploy mon create-initial

and let me know if the problem still persists.

Actually it is not able to find "client.admin.keyring" file as it has not been created yet.

Thanks 

Ashish Chandra

On Fri, Feb 14, 2014 at 6:15 PM, Ron Gage <ron@xxxxxxxxxxx> wrote:

Greetings all!

I am trying to deploy a proof of concept system for a replicated fault tolerant CIFS server cluster using CEPH as the back end.  I am having significant trouble getting the CEPH part to install correctly.  Specifically, I can't get the monitor to come up into a useful state due to the apparent lack of a keyring or two.  Here is the background:

 

Everything is running on a standalone VMware server with internal storage.  The idea here is to see it work, not to see how well it performs.

 

Current configuraton: 4 x OSD, 1 x monitor/manager

OSD: 2 x CPU, 8 gig memory, internal only networking, 16 gig sda for boot/root, 160 gig sdb for CEPH store, Centos 6.5 x64 fully up to date, minimal install

mon/mgr: 2 x CPU, 2 gig memory, internal and external networking, 16 gig sda, Centos 6.5 x64 fully up to date, minimal install

 

Since I have made several attempts already to get this working, I execute the following commands from the mgr to reset everything (logged in as user ceph)

ceph-deploy purgedata c1 c2 c3 c4 cm

ceph-deploy forgetkeys

ceph-deploy purge c1 c2 c3 c4 cm

sudo rm -f /etc/ceph/*

sudo rm -f /var/lib/ceph/mon/*.*

cd ~/my-cluster

rm -f *

 

Now to deploy a straight forward single monitor node:

ceph-deploy new cm

 

And to add the public interface to the ceph.conf

vi ceph.conf

(add to end of global section) public network =  172.24.0.0/16

 

Now to install the ceph software in the 5 nodes:

ceph-deploy install cm c1 c2 c3 c4

(waiting...waiting...waiting...)

 

And here is where the problems begin - creating the initial monitor:

[ceph@cm my-cluster]$ ceph-deploy mon create-initial
[ceph_deploy.cli][INFO  ] Invoked (1.3.5): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cm
[ceph_deploy.mon][DEBUG ] detecting platform for host cm ...
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: CentOS 6.5 Final
[cm][DEBUG ] determining if provided host has same hostname in remote
[cm][DEBUG ] get remote short hostname
[cm][DEBUG ] deploying mon to cm
[cm][DEBUG ] get remote short hostname
[cm][DEBUG ] remote hostname: cm
[cm][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cm][DEBUG ] create the mon path if it does not exist
[cm][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cm/done
[cm][DEBUG ] create a done file to avoid re-doing the mon deployment
[cm][DEBUG ] create the init path if it does not exist
[cm][DEBUG ] locating the `service` executable...
[cm][INFO  ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cm
[cm][DEBUG ] === mon.cm ===
[cm][DEBUG ] Starting Ceph mon.cm on cm...
[cm][DEBUG ] Starting ceph-create-keys on cm...
[cm][WARNIN] No data was received after 7 seconds, disconnecting...
[cm][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status
[cm][DEBUG ] ********************************************************************************
[cm][DEBUG ] status for monitor: mon.cm
[cm][DEBUG ] {
[cm][DEBUG ]   "election_epoch": 1,
[cm][DEBUG ]   "extra_probe_peers": [],
[cm][DEBUG ]   "monmap": {
[cm][DEBUG ]     "created": "0.000000",
[cm][DEBUG ]     "epoch": 1,
[cm][DEBUG ]     "fsid": "b11875f0-e611-40b9-ae81-e7cba7586ed5",
[cm][DEBUG ]     "modified": "0.000000",
[cm][DEBUG ]     "mons": [
[cm][DEBUG ]       {
[cm][DEBUG ]         "addr": " 172.24.12.91:6789/0",
[cm][DEBUG ]         "name": "cm",
[cm][DEBUG ]         "rank": 0
[cm][DEBUG ]       }
[cm][DEBUG ]     ]
[cm][DEBUG ]   },
[cm][DEBUG ]   "name": "cm",
[cm][DEBUG ]   "outside_quorum": [],
[cm][DEBUG ]   "quorum": [
[cm][DEBUG ]     0
[cm][DEBUG ]   ],
[cm][DEBUG ]   "rank": 0,
[cm][DEBUG ]   "state": "leader",
[cm][DEBUG ]   "sync_provider": []
[cm][DEBUG ] }
[cm][DEBUG ] ********************************************************************************
[cm][INFO  ] monitor: mon.cm is running
[cm][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status
[ceph_deploy.mon][INFO  ] processing monitor mon.cm
[cm][DEBUG ] connected to host: cm
[cm][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status
[ceph_deploy.mon][INFO  ] mon.cm monitor has reached quorum!
[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO  ] Running gatherkeys...
[ceph_deploy.gatherkeys][DEBUG ] Checking cm for /etc/ceph/ceph.client.admin.keyring
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[cm][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['cm']
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking cm for /var/lib/ceph/bootstrap-osd/ceph.keyring
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[cm][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm']
[ceph_deploy.gatherkeys][DEBUG ] Checking cm for /var/lib/ceph/bootstrap-mds/ceph.keyring
[cm][DEBUG ] connected to host: cm
[cm][DEBUG ] detect platform information from remote host
[cm][DEBUG ] detect machine type
[cm][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm']
[ceph@cm my-cluster]$

 

 

From my untrained eye, it looks like several key files aren't either being created or copied to the expected place.

 

Contents of ~/my-cluster:

[ceph@cm my-cluster]$ ls
ceph.conf  ceph.log  ceph.mon.keyring  epel-release-6-8.noarch.rpm

 

Contents of /var/lib/ceph/mon/ceph-cm:

[ceph@cm ceph-cm]$ ls -R .: done  keyring  store.db  sysvinit

./store.db: 000016.sst  000021.sst  CURRENT  LOG 000018.sst  000022.log  LOCK     MANIFEST-000020

Contents of /etc/ceph:

[ceph@cm ceph]$ ls
ceph.client.admin.keyring.5704.tmp  ceph.conf  rbdmap

As you can plainly see, I have followed the directions in the "quick start" deployment guide on ceph.com (http://ceph.com/docs/master/start/quick-ceph-deploy/#create-a-cluster).

What am I missing?  Are the directions incomplete (or wrong)?

 

HELP!

 

 

Ron Gage
Westland, Michigan

Please be sure to check out My Photography Website

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

Ron Gage
Westland, Michigan

Please be sure to check out My Photography Website

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com