Re: Problems with keyrings during deployment

[Alfredo Deza <alfredo.deza@xxxxxxxxxxx>]

On Tue, Aug 27, 2013 at 12:30 PM, Francesc Alted <francesc@xxxxxxxxxxxx> wrote:

On Tue, Aug 27, 2013 at 6:25 PM, Alfredo Deza <alfredo.deza@xxxxxxxxxxx> wrote:

On Tue, Aug 27, 2013 at 12:04 PM, Francesc Alted <francesc@xxxxxxxxxxxx> wrote:

On Tue, Aug 27, 2013 at 5:29 PM, Sage Weil <sage@xxxxxxxxxxx> wrote:

On Tue, 27 Aug 2013, Francesc Alted wrote:
> Hi again,
>
> I continue to try debugging the problem reported before.  Now, I have been
> trying to use a couple of VM for doing this (one with Ubuntu 12.04 64-bit,
> and the other with Ubuntu 12.10 64-bit, and I use the ceph.com repos for
> installing the Ceph libraries).  And, unfortunately, I am getting into the
> same problem: the keyring do not appear where they should (i.e.
> bootstrap-mds and bootstrap-osd in /var/lib/ceph).
>
> I have followed the preflight check list
> (http://ceph.com/docs/next/start/quick-start-preflight/), and the ceph user
> in the admin box can login perfectly well on the server box, so not sure
> what's going on here.
>
> I have even tried to use a single ceph server for installing everything
> (adding the 'osd crush chooseleaf type = 0' line into the ceph conf file)
> but then again the keyrings do not appear.
>
> Nobody is having the same problems than me (using latest Ceph Dumpling
> 0.67.2 release here)? 
>
> Thanks for any insight!

There are several possible pitfalls here; the missing keys are just the
most visible symptom of the monitors not forming an initial quorum.

Can you post the contents of your ceph.conf and output from 'ceph daemon
mon.`hostnam` mon-status' on each of the mon nodes?

Okay, I tracked down my problem.  It turned out that I was setting different names for the ceph servers in /etc/hosts than their own `hostname`.  These log lines when creating the monitor gave me the clue:

[ceph-server2][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-vagrant.mon.keyring

[ceph-server2][INFO  ] create the monitor keyring file

[ceph-server2][INFO  ] Running command: ceph-mon --cluster ceph --mkfs -i vagrant --keyring /var/lib/ceph/tmp/ceph-vagrant.mon.keyring

[ceph-server2][INFO  ] ceph-mon: mon.noname-a 192.168.33.11:6789/0 is local, renaming to mon.vagrant

[ceph-server2][INFO  ] ceph-mon: set fsid to 253c5a74-699b-44ef-a071-5883716fa620

I was calling this 'vagrant' hostname 'ceph-server2' in my /etc/hosts and I realized this was fooling cephs.  So I changed all my /etc/hosts to follow the original hostnames (changed to 'quantal64'), and pum! everything works as intended:

[quantal64][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-quantal64.mon.keyring

[quantal64][INFO  ] create the monitor keyring file

[quantal64][INFO  ] Running command: ceph-mon --cluster ceph --mkfs -i quantal64 --keyring /var/lib/ceph/tmp/ceph-quantal64.mon.keyring

[quantal64][INFO  ] ceph-mon: mon.noname-a 192.168.33.11:6789/0 is local, renaming to mon.quantal64

[quantal64][INFO  ] ceph-mon: set fsid to 96c48ec5-7dd5-4f76-81f9-4fdc711a76f0

Now I can gather the keys normally:

$ ceph-deploy gatherkeys quantal64

[ceph_deploy.gatherkeys][DEBUG ] Checking quantal64 for /etc/ceph/ceph.client.admin.keyring

[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from quantal64.

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring

[ceph_deploy.gatherkeys][DEBUG ] Checking quantal64 for /var/lib/ceph/bootstrap-osd/ceph.keyring

[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from quantal64.

[ceph_deploy.gatherkeys][DEBUG ] Checking quantal64 for /var/lib/ceph/bootstrap-mds/ceph.keyring

[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from quantal64.

 

Well, thanks anyways.  Now it is time to make some more progress and create some ODSs :)

Francesc, thanks for pasting this log info, it is useful to know what worked for you :) I will update the docs for ceph-deploy

on things to watch out so that there is *something* users can try when this comes up.

No problem.  A possible idea for enhancing the capabilities to self-detecting problems would be to implement a check in ceph-deploy (or in another place) that warns (or just gives an error) when it detects that the hostname is different depending on whether they do a DNS lookup or a `hostname` output.

I went ahead and created http://tracker.ceph.com/issues/6132 to track this.

Thanks again.

--
Francesc Alted

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com