Re: ceph-deploy ceph-create-keys hangs

[Alfredo Deza <alfredo.deza@xxxxxxxxxxx>]

On Fri, Aug 2, 2013 at 9:17 AM, Mathias Lindberg <mathlin@xxxxxxxxxxx> wrote:

Hi, 

I have tried this with almost the same result, there seems to be one more set of processes. 

cloud-2: root     27782     1  0 14:10 ?        00:00:00 /usr/bin/ceph-mon -i cloud-2 --pid-file /var/run/ceph/mon.cloud-2.pid -c /etc/ceph/ceph.conf

cloud-2: root     27802     1  0 14:10 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-2

cloud-2: root     27824 27802  0 14:10 ?        00:00:00 ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-cloud-2/keyring auth get-or-create client.admin mon allow * osd allow * mds allow

cloud-1: root     31797     1  0 14:10 ?        00:00:00 /usr/bin/ceph-mon -i cloud-1 --pid-file /var/run/ceph/mon.cloud-1.pid -c /etc/ceph/ceph.conf

cloud-1: root     31815     1  0 14:10 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-1

cloud-1: root     31839 31815  0 14:10 ?        00:00:00 ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-cloud-1/keyring auth get-or-create client.admin mon allow * osd allow * mds allow

cloud-0: root     19629  8546  0 Aug01 pts/0    00:00:00 su - ceph

cloud-0: ceph     19630 19629  0 Aug01 pts/0    00:00:00 -bash

cloud-0: root     24090     1  0 14:10 pts/0    00:00:00 /usr/bin/ceph-mon -i cloud-0 --pid-file /var/run/ceph/mon.cloud-0.pid -c /etc/ceph/ceph.conf

cloud-0: root     24106     1  0 14:10 pts/0    00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-0

cloud-0: root     24143 24106  0 14:10 pts/0    00:00:00 ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-cloud-0/keyring auth get-or-create client.admin mon allow * osd allow * mds allow

cloud-0: ceph     24793 19630  0 14:25 pts/0    00:00:00 grep ceph

Regards,
Mathias

On Aug 1, 2013, at 18:05 , Alfredo Deza <alfredo.deza@xxxxxxxxxxx> wrote:

Hi Mathias,

Have you tried these steps and sticking to the slow interfaces? I would be curious to see if this is just a problem of how those interfaces are able to talk to each other.

 

From: Mathias Lindberg <mathlin@xxxxxxxxxxx>
Date: August 1, 2013, 4:01:38 MDT
To: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
Subject: ceph-deploy ceph-create-keys hangs

Hi

Having previously had problems during startup with "creating keys" (otherwise a working setup) one one node when using mkcephfs, i have given ceph-deploy a try and get stuck on what feels like the same step.

Ceph version is 0.61.7 and OS is centos 6.4.

Steps i have done is:

#ceph-deploy new cloud-{0,1,2}-fast

#ceph-deploy --overwrite-conf mon create cloud-{0,1,2}-fast 

Ceph-mon starts ok on all nodes and ceph-create-keys seems to be stuck.

cloud-1-fast: root      6580     1  0 11:45 ?        00:00:00 /usr/bin/ceph-mon -i cloud-1 --pid-file /var/run/ceph/mon.cloud-1.pid -c /etc/ceph/ceph.conf

cloud-1-fast: root      6601     1  0 11:45 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-1

cloud-2-fast: root     18724     1  0 11:45 ?        00:00:00 /usr/bin/ceph-mon -i cloud-2 --pid-file /var/run/ceph/mon.cloud-2.pid -c /etc/ceph/ceph.conf

cloud-2-fast: root     18747     1  0 11:45 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-2

cloud-0-fast: root     19629  8546  0 11:44 pts/0    00:00:00 su - ceph

cloud-0-fast: ceph     19630 19629  0 11:44 pts/0    00:00:00 -bash

cloud-0-fast: root     19853     1  0 11:45 ?        00:00:00 /usr/bin/ceph-mon -i cloud-0 --pid-file /var/run/ceph/mon.cloud-0.pid -c /etc/ceph/ceph.conf

cloud-0-fast: root     19872     1  0 11:45 ?        00:00:00 /usr/bin/python /usr/sbin/ceph-create-keys -i cloud-0

cloud-0-fast: ceph     20282 19630  0 11:46 pts/0    00:00:00 grep ceph

I have the /var/log/ceph/ceph-mon.cloud-0.log with debug turned on at http://pastebin.com/DivE95mK 

And the output from a strace of the ceph-create-keys from one of the nodes at http://pastebin.com/JQak151Z

Worth mentioning is that each server has 2 interfaces cloud-* ("normal interface") and cloud-*-fast (10gE that i want to use), hostname resolves to cloud-*.

The small ceph.conf file created so far

[global]

debug_ms = 1

filestore_xattr_use_omap = true

debug_monc = 20

mon_host = 10.12.1.160,10.12.1.161,10.12.1.162

osd_journal_size = 1024

debug_mon = 20

mon_initial_members = cloud-0-fast, cloud-1-fast, cloud-2-fast

auth_supported = cephx

fsid = 90578caa-3c63-4183-96c7-176467a98ddb

Regards,

Mathias

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Have you tried running the create keys command manually in the host that is failing?

What is the output of:

    ceph-create-keys -i cloud-0

Replace cloud-0 with whatever node you are using.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com