|
Greetings all!
I am trying to deploy a proof of concept system for a replicated fault tolerant CIFS server cluster using CEPH as the back end. I am having significant trouble getting the CEPH part to install correctly. Specifically, I can't get the monitor to come up into a useful state due to the apparent lack of a keyring or two. Here is the background:
Everything is running on a standalone VMware server with internal storage. The idea here is to see it work, not to see how well it performs.
Current configuraton: 4 x OSD, 1 x monitor/manager
OSD: 2 x CPU, 8 gig memory, internal only networking, 16 gig sda for boot/root, 160 gig sdb for CEPH store, Centos 6.5 x64 fully up to date, minimal install
mon/mgr: 2 x CPU, 2 gig memory, internal and external networking, 16 gig sda, Centos 6.5 x64 fully up to date, minimal install
Since I have made several attempts already to get this working, I execute the following commands from the mgr to reset everything (logged in as user ceph)
ceph-deploy purgedata c1 c2 c3 c4 cm
ceph-deploy forgetkeys
ceph-deploy purge c1 c2 c3 c4 cm
sudo rm -f /etc/ceph/*
sudo rm -f /var/lib/ceph/mon/*.*
cd ~/my-cluster
rm -f *
Now to deploy a straight forward single monitor node:
ceph-deploy new cm
And to add the public interface to the ceph.conf
vi ceph.conf
(add to end of global section) public network = 172.24.0.0/16
Now to install the ceph software in the 5 nodes:
ceph-deploy install cm c1 c2 c3 c4
(waiting...waiting...waiting...)
And here is where the problems begin - creating the initial monitor:
[ceph@cm my-cluster]$ ceph-deploy mon create-initial
[ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon create-initial [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cm [ceph_deploy.mon][DEBUG ] detecting platform for host cm ... [cm][DEBUG ] connected to host: cm [cm][DEBUG ] detect platform information from remote host [cm][DEBUG ] detect machine type [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final [cm][DEBUG ] determining if provided host has same hostname in remote [cm][DEBUG ] get remote short hostname [cm][DEBUG ] deploying mon to cm [cm][DEBUG ] get remote short hostname [cm][DEBUG ] remote hostname: cm [cm][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [cm][DEBUG ] create the mon path if it does not exist [cm][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cm/done [cm][DEBUG ] create a done file to avoid re-doing the mon deployment [cm][DEBUG ] create the init path if it does not exist [cm][DEBUG ] locating the `service` executable... [cm][INFO ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cm [cm][DEBUG ] === mon.cm === [cm][DEBUG ] Starting Ceph mon.cm on cm... [cm][DEBUG ] Starting ceph-create-keys on cm... [cm][WARNIN] No data was received after 7 seconds, disconnecting... [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status [cm][DEBUG ] ******************************************************************************** [cm][DEBUG ] status for monitor: mon.cm [cm][DEBUG ] { [cm][DEBUG ] "election_epoch": 1, [cm][DEBUG ] "extra_probe_peers": [], [cm][DEBUG ] "monmap": { [cm][DEBUG ] "created": "0.000000", [cm][DEBUG ] "epoch": 1, [cm][DEBUG ] "fsid": "b11875f0-e611-40b9-ae81-e7cba7586ed5", [cm][DEBUG ] "modified": "0.000000", [cm][DEBUG ] "mons": [ [cm][DEBUG ] { [cm][DEBUG ] "addr": "172.24.12.91:6789/0", [cm][DEBUG ] "name": "cm", [cm][DEBUG ] "rank": 0 [cm][DEBUG ] } [cm][DEBUG ] ] [cm][DEBUG ] }, [cm][DEBUG ] "name": "cm", [cm][DEBUG ] "outside_quorum": [], [cm][DEBUG ] "quorum": [ [cm][DEBUG ] 0 [cm][DEBUG ] ], [cm][DEBUG ] "rank": 0, [cm][DEBUG ] "state": "leader", [cm][DEBUG ] "sync_provider": [] [cm][DEBUG ] } [cm][DEBUG ] ******************************************************************************** [cm][INFO ] monitor: mon.cm is running [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status [ceph_deploy.mon][INFO ] processing monitor mon.cm [cm][DEBUG ] connected to host: cm [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cm.asok mon_status [ceph_deploy.mon][INFO ] mon.cm monitor has reached quorum! [ceph_deploy.mon][INFO ] all initial monitors are running and have formed quorum [ceph_deploy.mon][INFO ] Running gatherkeys... [ceph_deploy.gatherkeys][DEBUG ] Checking cm for /etc/ceph/ceph.client.admin.keyring [cm][DEBUG ] connected to host: cm [cm][DEBUG ] detect platform information from remote host [cm][DEBUG ] detect machine type [cm][DEBUG ] fetch remote file [ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['cm'] [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring [ceph_deploy.gatherkeys][DEBUG ] Checking cm for /var/lib/ceph/bootstrap-osd/ceph.keyring [cm][DEBUG ] connected to host: cm [cm][DEBUG ] detect platform information from remote host [cm][DEBUG ] detect machine type [cm][DEBUG ] fetch remote file [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm'] [ceph_deploy.gatherkeys][DEBUG ] Checking cm for /var/lib/ceph/bootstrap-mds/ceph.keyring [cm][DEBUG ] connected to host: cm [cm][DEBUG ] detect platform information from remote host [cm][DEBUG ] detect machine type [cm][DEBUG ] fetch remote file [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm'] [ceph@cm my-cluster]$
From my untrained eye, it looks like several key files aren't either being created or copied to the expected place.
Contents of ~/my-cluster:
[ceph@cm my-cluster]$ ls
ceph.conf ceph.log ceph.mon.keyring epel-release-6-8.noarch.rpm
Contents of /var/lib/ceph/mon/ceph-cm:
[ceph@cm ceph-cm]$ ls -R .: done keyring store.db sysvinit ./store.db: 000016.sst 000021.sst CURRENT LOG 000018.sst 000022.log LOCK MANIFEST-000020 Contents of /etc/ceph: [ceph@cm ceph]$ ls As you can plainly see, I have followed the directions in the "quick start" deployment guide on ceph.com (http://ceph.com/docs/master/start/quick-ceph-deploy/#create-a-cluster). What am I missing? Are the directions incomplete (or wrong)?
HELP! |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
