Re: ceph-deploy ceph-create-keys hangs

Mathias Lindberg <mathlin@xxxxxxxxxxx> · Fri, 2 Aug 2013 13:17:04 +0000

Hi, 
I have tried this with almost the same result, there seems to be one more set of processes. 

cloud-2: root     27782     1  0 14:10 ?        00:00:00 /usr/bin/ceph-mon -i cloud-2 --pid-file /var/run/ceph/mon.cloud-2.pid -c /etc/ceph/ceph.conf
cloud-2: root     27802     1  0 14:10 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-2
cloud-2: root     27824 27802  0 14:10 ?        00:00:00 ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-cloud-2/keyring auth get-or-create client.admin mon allow * osd allow * mds allow
cloud-1: root     31797     1  0 14:10 ?        00:00:00 /usr/bin/ceph-mon -i cloud-1 --pid-file /var/run/ceph/mon.cloud-1.pid -c /etc/ceph/ceph.conf
cloud-1: root     31815     1  0 14:10 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-1
cloud-1: root     31839 31815  0 14:10 ?        00:00:00 ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-cloud-1/keyring auth get-or-create client.admin mon allow * osd allow * mds allow
cloud-0: root     19629  8546  0 Aug01 pts/0    00:00:00 su - ceph
cloud-0: ceph     19630 19629  0 Aug01 pts/0    00:00:00 -bash
cloud-0: root     24090     1  0 14:10 pts/0    00:00:00 /usr/bin/ceph-mon -i cloud-0 --pid-file /var/run/ceph/mon.cloud-0.pid -c /etc/ceph/ceph.conf
cloud-0: root     24106     1  0 14:10 pts/0    00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-0
cloud-0: root     24143 24106  0 14:10 pts/0    00:00:00 ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-cloud-0/keyring auth get-or-create client.admin mon allow * osd allow * mds allow
cloud-0: ceph     24793 19630  0 14:25 pts/0    00:00:00 grep ceph

Regards,
Mathias

On Aug 1, 2013, at 18:05 , Alfredo Deza <alfredo.deza@xxxxxxxxxxx> wrote:

Hi Mathias,

Have you tried these steps and sticking to the slow interfaces? I would be curious to see if this is just a problem of how those interfaces are able to talk to each other.

From: Mathias Lindberg <mathlin@xxxxxxxxxxx>

Date: August 1, 2013, 4:01:38 MDT
To: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>

Subject:  ceph-deploy ceph-create-keys hangs

Hi
Having previously had problems during startup with "creating keys" (otherwise a working setup) one one node when using mkcephfs, i have given ceph-deploy a try and get stuck on what feels like the same step.
Ceph version is 0.61.7 and OS is centos 6.4.

Steps i have done is:
#ceph-deploy new cloud-{0,1,2}-fast
#ceph-deploy --overwrite-conf mon create cloud-{0,1,2}-fast 

Ceph-mon starts ok on all nodes and ceph-create-keys seems to be stuck.

cloud-1-fast: root      6580     1  0 11:45 ?        00:00:00 /usr/bin/ceph-mon -i cloud-1 --pid-file /var/run/ceph/mon.cloud-1.pid -c /etc/ceph/ceph.conf
cloud-1-fast: root      6601     1  0 11:45 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-1
cloud-2-fast: root     18724     1  0 11:45 ?        00:00:00 /usr/bin/ceph-mon -i cloud-2 --pid-file /var/run/ceph/mon.cloud-2.pid -c /etc/ceph/ceph.conf
cloud-2-fast: root     18747     1  0 11:45 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-2
cloud-0-fast: root     19629  8546  0 11:44 pts/0    00:00:00 su - ceph
cloud-0-fast: ceph     19630 19629  0 11:44 pts/0    00:00:00 -bash
cloud-0-fast: root     19853     1  0 11:45 ?        00:00:00 /usr/bin/ceph-mon -i cloud-0 --pid-file /var/run/ceph/mon.cloud-0.pid -c /etc/ceph/ceph.conf
cloud-0-fast: root     19872     1  0 11:45 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-0
cloud-0-fast: ceph     20282 19630  0 11:46 pts/0    00:00:00 grep ceph

I have the /var/log/ceph/ceph-mon.cloud-0.log with debug turned on at http://pastebin.com/DivE95mK 
And the output from a strace of the ceph-create-keys from one of the nodes at http://pastebin.com/JQak151Z

Worth mentioning is that each server has 2 interfaces cloud-* ("normal interface") and cloud-*-fast (10gE that i want to use), hostname resolves to cloud-*.
The small ceph.conf file created so far

[global]
debug_ms = 1
filestore_xattr_use_omap = true
debug_monc = 20
mon_host = 10.12.1.160,10.12.1.161,10.12.1.162
osd_journal_size = 1024
debug_mon = 20
mon_initial_members = cloud-0-fast, cloud-1-fast, cloud-2-fast
auth_supported = cephx
fsid = 90578caa-3c63-4183-96c7-176467a98ddb

Regards,
Mathias

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Attachment:
smime.p7s

Description: S/MIME cryptographic signature
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com