On Fri, Feb 14, 2014 at 10:14 AM, Ron Gage <ron@xxxxxxxxxxx> wrote:
> Alfredo:
> As near as I can tell, ceph-create-keys is hanging - or taking a huge amount
> ot fime to finish.
Correct! Do you have iptables enabled by any chance?
Also it can be useful if you manually execute that create-keys command
and see the output.
>
> Segment from screen output:
> (Make sure that ALL ceph programs/processes are not running)
>
> [ceph@cm my-cluster]$ ceph-deploy mon create-initial cm
> ...
> [cm][DEBUG ] === mon.cm ===
> [cm][DEBUG ] Starting Ceph mon.cm on cm...
> [cm][DEBUG ] Starting ceph-create-keys on cm...
> [cm][WARNIN] No data was received after 7 seconds, disconnecting...
> [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
> /var/run/ceph/ceph-mon.cm.asok mon_status
> ...
>
> [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
> /etc/ceph/ceph.client.admin.keyring
> [cm][DEBUG ] connected to host: cm
> [cm][DEBUG ] detect platform information from remote host
> [cm][DEBUG ] detect machine type
> [cm][DEBUG ] fetch remote file
> [ceph_deploy.gatherkeys][WARNIN] Unable to find
> /etc/ceph/ceph.client.admin.keyring on ['cm']
> [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
> [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
> /var/lib/ceph/bootstrap-osd/ceph.keyring
> [cm][DEBUG ] connected to host: cm
> [cm][DEBUG ] detect platform information from remote host
> [cm][DEBUG ] detect machine type
> [cm][DEBUG ] fetch remote file
> [ceph_deploy.gatherkeys][WARNIN] Unable to find
> /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm']
> [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
> /var/lib/ceph/bootstrap-mds/ceph.keyring
> [cm][DEBUG ] connected to host: cm
> [cm][DEBUG ] detect platform information from remote host
> [cm][DEBUG ] detect machine type
> [cm][DEBUG ] fetch remote file
> [ceph_deploy.gatherkeys][WARNIN] Unable to find
> /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm']
>
> [ceph@cm my-cluster]$ ps ax | grep -i ceph
> 1298 ? Ss 0:00 sshd: ceph [priv]
> 1302 ? S 0:00 sshd: ceph@pts/0
> 2058 pts/0 Sl 0:00 /usr/bin/ceph-mon -i cm --pid-file
> /var/run/ceph/mon.cm.pid -c /etc/ceph/ceph.conf
> 2071 pts/0 S 0:00 python /usr/sbin/ceph-create-keys -i cm
> 2081 pts/0 Sl 0:00 python /usr/bin/ceph --cluster=ceph --name=mon.
> --keyring=/var/lib/ceph/mon/ceph-cm/keyring auth get-or-create client.admin
> mon allow * osd allow * mds allow
> 2212 pts/0 S+ 0:00 grep -i ceph
> [ceph@cm my-cluster]$
>
>
>
>> On February 14, 2014 at 9:41 AM Alfredo Deza <alfredo.deza@xxxxxxxxxxx>
>> wrote:
>>
>>
>> On Fri, Feb 14, 2014 at 9:38 AM, Ron Gage <ron@xxxxxxxxxxx> wrote:
>> > Thanks Alfredo!
>> > Would that be /etc/ceph/ceph.conf or ~/my-cluster/ceph.conf
>>
>> Good question, sorry for the ambiguity. From where you are running
>> ceph-deploy, you should have a ceph.conf there, you can edit
>> that one file and then call 'ceph-deploy config push {nodes}' where
>> {nodes} should be replaced with the server(s) to push the config to.
>>
>>
>> >
>> > Ron Gage
>> >
>> >
>> >> On February 14, 2014 at 9:31 AM Alfredo Deza <alfredo.deza@xxxxxxxxxxx>
>> >> wrote:
>> >>
>> >>
>> >> On Fri, Feb 14, 2014 at 9:08 AM, Ron Gage <ron@xxxxxxxxxxx> wrote:
>> >> > Please see the post you replied to. The admin keyring is not being
>> >> > created.
>> >> >
>> >> > Thanks!
>> >> >
>> >> > Ron
>> >>
>> >> There is no need to run other commands instead of create-initial. That
>> >> is the best way to get started with monitor deployment.
>> >>
>> >> From ceph-deploy's perspective, everything has been called correctly
>> >> and the output from the node looks correct. The next thing I would
>> >> suggest is to bump the log levels of the monitors in ceph.conf and
>> >> restart them so that you can inspect a bit better what is going on
>> >> there.
>> >>
>> >> In the [global] section of ceph.conf:
>> >>
>> >> mon debug = 10
>> >>
>> >>
>> >>
>> >> >
>> >> >
>> >> > On February 14, 2014 at 8:54 AM Srinivasa Rao Ragolu
>> >> > <sragolu@xxxxxxxxxx>
>> >> > wrote:
>> >> >
>> >> > Follow the below link as it is
>> >> >
>> >> > http://karan-mj.blogspot.fi/2013/12/ceph-installation-part-2.html
>> >> >
>> >> > Thanks,
>> >> > Srinivas.
>> >> >
>> >> >
>> >> > On Fri, Feb 14, 2014 at 7:21 PM, Ron Gage <ron@xxxxxxxxxxx> wrote:
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > [ceph@cm my-cluster]$ ceph-deploy mon create cm
>> >> > [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon
>> >> > create
>> >> > cm
>> >> > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cm
>> >> > [ceph_deploy.mon][DEBUG ] detecting platform for host cm ...
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
>> >> > [cm][DEBUG ] determining if provided host has same hostname in remote
>> >> > [cm][DEBUG ] get remote short hostname
>> >> > [cm][DEBUG ] deploying mon to cm
>> >> > [cm][DEBUG ] get remote short hostname
>> >> > [cm][DEBUG ] remote hostname: cm
>> >> > [cm][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>> >> > [cm][DEBUG ] create the mon path if it does not exist
>> >> > [cm][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cm/done
>> >> > [cm][DEBUG ] create a done file to avoid re-doing the mon deployment
>> >> > [cm][DEBUG ] create the init path if it does not exist
>> >> > [cm][DEBUG ] locating the `service` executable...
>> >> > [cm][INFO ] Running command: sudo /sbin/service ceph -c
>> >> > /etc/ceph/ceph.conf
>> >> > start mon.cm
>> >> > [cm][DEBUG ] === mon.cm ===
>> >> > [cm][DEBUG ] Starting Ceph mon.cm on cm...
>> >> > [cm][DEBUG ] Starting ceph-create-keys on cm...
>> >> > [cm][WARNIN] No data was received after 7 seconds, disconnecting...
>> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>> >> > /var/run/ceph/ceph-mon.cm.asok mon_status
>> >> > [cm][DEBUG ]
>> >> >
>> >> >
>> >> > ********************************************************************************
>> >> > [cm][DEBUG ] status for monitor: mon.cm
>> >> > [cm][DEBUG ] {
>> >> > [cm][DEBUG ] "election_epoch": 1,
>> >> > [cm][DEBUG ] "extra_probe_peers": [],
>> >> > [cm][DEBUG ] "monmap": {
>> >> > [cm][DEBUG ] "created": "0.000000",
>> >> > [cm][DEBUG ] "epoch": 1,
>> >> > [cm][DEBUG ] "fsid": "b11875f0-e611-40b9-ae81-e7cba7586ed5",
>> >> > [cm][DEBUG ] "modified": "0.000000",
>> >> > [cm][DEBUG ] "mons": [
>> >> > [cm][DEBUG ] {
>> >> > [cm][DEBUG ] "addr": " 172.24.12.91:6789/0",
>> >> > [cm][DEBUG ] "name": "cm",
>> >> > [cm][DEBUG ] "rank": 0
>> >> > [cm][DEBUG ] }
>> >> > [cm][DEBUG ] ]
>> >> > [cm][DEBUG ] },
>> >> > [cm][DEBUG ] "name": "cm",
>> >> > [cm][DEBUG ] "outside_quorum": [],
>> >> > [cm][DEBUG ] "quorum": [
>> >> > [cm][DEBUG ] 0
>> >> > [cm][DEBUG ] ],
>> >> > [cm][DEBUG ] "rank": 0,
>> >> > [cm][DEBUG ] "state": "leader",
>> >> > [cm][DEBUG ] "sync_provider": []
>> >> > [cm][DEBUG ] }
>> >> > [cm][DEBUG ]
>> >> >
>> >> >
>> >> > ********************************************************************************
>> >> > [cm][INFO ] monitor: mon.cm is running
>> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>> >> > /var/run/ceph/ceph-mon.cm.asok mon_status
>> >> > [ceph@cm my-cluster]$ ^C
>> >> >
>> >> > [ceph@cm my-cluster]$ ceph-deploy gatherkeys cm
>> >> > [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy
>> >> > gatherkeys
>> >> > cm
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
>> >> > /etc/ceph/ceph.client.admin.keyring
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [cm][DEBUG ] fetch remote file
>> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find
>> >> > /etc/ceph/ceph.client.admin.keyring on ['cm']
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
>> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [cm][DEBUG ] fetch remote file
>> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find
>> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm']
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
>> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [cm][DEBUG ] fetch remote file
>> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find
>> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm']
>> >> > [ceph@cm my-cluster]$
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On February 14, 2014 at 8:20 AM Ashish Chandra <
>> >> > mail.ashishchandra@xxxxxxxxx> wrote:
>> >> >
>> >> > Hi Ron,
>> >> > Can you try running :
>> >> >
>> >> > ceph-deploy mon create {ceph-node}
>> >> >
>> >> > ceph-deploy gatherkeys {ceph-node}
>> >> >
>> >> >
>> >> > as two different commands, instead of running
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ceph-deploy mon create-initial
>> >> >
>> >> > and let me know if the problem still persists.
>> >> >
>> >> > Actually it is not able to find "client.admin.keyring" file as it has
>> >> > not
>> >> > been created yet.
>> >> >
>> >> > Thanks
>> >> >
>> >> > Ashish Chandra
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Feb 14, 2014 at 6:15 PM, Ron Gage <ron@xxxxxxxxxxx> wrote:
>> >> >
>> >> > Greetings all!
>> >> > I am trying to deploy a proof of concept system for a replicated
>> >> > fault
>> >> > tolerant CIFS server cluster using CEPH as the back end. I am having
>> >> > significant trouble getting the CEPH part to install correctly.
>> >> > Specifically, I can't get the monitor to come up into a useful state
>> >> > due
>> >> > to
>> >> > the apparent lack of a keyring or two. Here is the background:
>> >> >
>> >> > Everything is running on a standalone VMware server with internal
>> >> > storage.
>> >> > The idea here is to see it work, not to see how well it performs.
>> >> >
>> >> > Current configuraton: 4 x OSD, 1 x monitor/manager
>> >> > OSD: 2 x CPU, 8 gig memory, internal only networking, 16 gig sda for
>> >> > boot/root, 160 gig sdb for CEPH store, Centos 6.5 x64 fully up to
>> >> > date,
>> >> > minimal install
>> >> > mon/mgr: 2 x CPU, 2 gig memory, internal and external networking, 16
>> >> > gig
>> >> > sda, Centos 6.5 x64 fully up to date, minimal install
>> >> >
>> >> > Since I have made several attempts already to get this working, I
>> >> > execute
>> >> > the following commands from the mgr to reset everything (logged in as
>> >> > user
>> >> > ceph)
>> >> > ceph-deploy purgedata c1 c2 c3 c4 cm
>> >> > ceph-deploy forgetkeys
>> >> > ceph-deploy purge c1 c2 c3 c4 cm
>> >> > sudo rm -f /etc/ceph/*
>> >> > sudo rm -f /var/lib/ceph/mon/*.*
>> >> > cd ~/my-cluster
>> >> > rm -f *
>> >> >
>> >> > Now to deploy a straight forward single monitor node:
>> >> > ceph-deploy new cm
>> >> >
>> >> > And to add the public interface to the ceph.conf
>> >> > vi ceph.conf
>> >> > (add to end of global section) public network = 172.24.0.0/16
>> >> >
>> >> > Now to install the ceph software in the 5 nodes:
>> >> > ceph-deploy install cm c1 c2 c3 c4
>> >> > (waiting...waiting...waiting...)
>> >> >
>> >> > And here is where the problems begin - creating the initial monitor:
>> >> > [ceph@cm my-cluster]$ ceph-deploy mon create-initial
>> >> > [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon
>> >> > create-initial
>> >> > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cm
>> >> > [ceph_deploy.mon][DEBUG ] detecting platform for host cm ...
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
>> >> > [cm][DEBUG ] determining if provided host has same hostname in remote
>> >> > [cm][DEBUG ] get remote short hostname
>> >> > [cm][DEBUG ] deploying mon to cm
>> >> > [cm][DEBUG ] get remote short hostname
>> >> > [cm][DEBUG ] remote hostname: cm
>> >> > [cm][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>> >> > [cm][DEBUG ] create the mon path if it does not exist
>> >> > [cm][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cm/done
>> >> > [cm][DEBUG ] create a done file to avoid re-doing the mon deployment
>> >> > [cm][DEBUG ] create the init path if it does not exist
>> >> > [cm][DEBUG ] locating the `service` executable...
>> >> > [cm][INFO ] Running command: sudo /sbin/service ceph -c
>> >> > /etc/ceph/ceph.conf
>> >> > start mon.cm
>> >> > [cm][DEBUG ] === mon.cm ===
>> >> > [cm][DEBUG ] Starting Ceph mon.cm on cm...
>> >> > [cm][DEBUG ] Starting ceph-create-keys on cm...
>> >> > [cm][WARNIN] No data was received after 7 seconds, disconnecting...
>> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>> >> > /var/run/ceph/ceph-mon.cm.asok mon_status
>> >> > [cm][DEBUG ]
>> >> >
>> >> >
>> >> > ********************************************************************************
>> >> > [cm][DEBUG ] status for monitor: mon.cm
>> >> > [cm][DEBUG ] {
>> >> > [cm][DEBUG ] "election_epoch": 1,
>> >> > [cm][DEBUG ] "extra_probe_peers": [],
>> >> > [cm][DEBUG ] "monmap": {
>> >> > [cm][DEBUG ] "created": "0.000000",
>> >> > [cm][DEBUG ] "epoch": 1,
>> >> > [cm][DEBUG ] "fsid": "b11875f0-e611-40b9-ae81-e7cba7586ed5",
>> >> > [cm][DEBUG ] "modified": "0.000000",
>> >> > [cm][DEBUG ] "mons": [
>> >> > [cm][DEBUG ] {
>> >> > [cm][DEBUG ] "addr": " 172.24.12.91:6789/0",
>> >> > [cm][DEBUG ] "name": "cm",
>> >> > [cm][DEBUG ] "rank": 0
>> >> > [cm][DEBUG ] }
>> >> > [cm][DEBUG ] ]
>> >> > [cm][DEBUG ] },
>> >> > [cm][DEBUG ] "name": "cm",
>> >> > [cm][DEBUG ] "outside_quorum": [],
>> >> > [cm][DEBUG ] "quorum": [
>> >> > [cm][DEBUG ] 0
>> >> > [cm][DEBUG ] ],
>> >> > [cm][DEBUG ] "rank": 0,
>> >> > [cm][DEBUG ] "state": "leader",
>> >> > [cm][DEBUG ] "sync_provider": []
>> >> > [cm][DEBUG ] }
>> >> > [cm][DEBUG ]
>> >> >
>> >> >
>> >> > ********************************************************************************
>> >> > [cm][INFO ] monitor: mon.cm is running
>> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>> >> > /var/run/ceph/ceph-mon.cm.asok mon_status
>> >> > [ceph_deploy.mon][INFO ] processing monitor mon.cm
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon
>> >> > /var/run/ceph/ceph-mon.cm.asok mon_status
>> >> > [ceph_deploy.mon][INFO ] mon.cm monitor has reached quorum!
>> >> > [ceph_deploy.mon][INFO ] all initial monitors are running and have
>> >> > formed
>> >> > quorum
>> >> > [ceph_deploy.mon][INFO ] Running gatherkeys...
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
>> >> > /etc/ceph/ceph.client.admin.keyring
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [cm][DEBUG ] fetch remote file
>> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find
>> >> > /etc/ceph/ceph.client.admin.keyring on ['cm']
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
>> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [cm][DEBUG ] fetch remote file
>> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find
>> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm']
>> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for
>> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring
>> >> > [cm][DEBUG ] connected to host: cm
>> >> > [cm][DEBUG ] detect platform information from remote host
>> >> > [cm][DEBUG ] detect machine type
>> >> > [cm][DEBUG ] fetch remote file
>> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find
>> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm']
>> >> > [ceph@cm my-cluster]$
>> >> >
>> >> >
>> >> > From my untrained eye, it looks like several key files aren't either
>> >> > being
>> >> > created or copied to the expected place.
>> >> >
>> >> > Contents of ~/my-cluster:
>> >> > [ceph@cm my-cluster]$ ls
>> >> > ceph.conf ceph.log ceph.mon.keyring epel-release-6-8.noarch.rpm
>> >> >
>> >> > Contents of /var/lib/ceph/mon/ceph-cm:
>> >> >
>> >> > [ceph@cm ceph-cm]$ ls -R .: done keyring store.db sysvinit
>> >> >
>> >> > ./store.db: 000016.sst 000021.sst CURRENT LOG 000018.sst 000022.log
>> >> > LOCK MANIFEST-000020
>> >> >
>> >> > Contents of /etc/ceph:
>> >> >
>> >> > [ceph@cm ceph]$ ls
>> >> > ceph.client.admin.keyring.5704.tmp ceph.conf rbdmap
>> >> >
>> >> > As you can plainly see, I have followed the directions in the "quick
>> >> > start"
>> >> > deployment guide on ceph.com
>> >> >
>> >> > (http://ceph.com/docs/master/start/quick-ceph-deploy/#create-a-cluster).
>> >> >
>> >> > What am I missing? Are the directions incomplete (or wrong)?
>> >> >
>> >> >
>> >> >
>> >> > HELP!
>> >> >
>> >> >
>> >> >
>> >> > Ron Gage
>> >> > Westland, Michigan
>> >> >
>> >> > Please be sure to check out My Photography Website
>> >> >
>> >> > _______________________________________________
>> >> > ceph-users mailing list
>> >> > ceph-users@xxxxxxxxxxxxxx
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>> >> >
>> >> >
>> >> > Ron Gage
>> >> > Westland, Michigan
>> >> >
>> >> > Please be sure to check out My Photography Website
>> >> >
>> >> > _______________________________________________
>> >> > ceph-users mailing list
>> >> > ceph-users@xxxxxxxxxxxxxx
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>> >> >
>> >> >
>> >> > Ron Gage
>> >> > Westland, Michigan
>> >> >
>> >> > Please be sure to check out My Photography Website
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > ceph-users mailing list
>> >> > ceph-users@xxxxxxxxxxxxxx
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>> > Ron Gage
>> > Westland, Michigan
>> >
>> > Please be sure to check out My Photography Website
>> >
> Ron Gage
> Westland, Michigan
>
> Please be sure to check out My Photography Website
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
