Okay I created mkdir'ed /etc/ceph on the admin node and issued the command again and now it seems to go through: [ceph@ceph-admin-node-centos-6-4 mycluster]$ ceph-deploy admin ceph-admin-node-centos-6-4 ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy admin ceph-admin-node-centos-6-4 ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-admin-node-centos-6-4 [ceph_deploy.sudo_pushy][DEBUG ] will use a local connection with sudo [ceph_deploy.sudo_pushy][DEBUG ] will use a local connection with sudo [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node1-mon-centos-6-4 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node2-osd0-centos-6-4 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node3-osd1-centos-6-4 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph@ceph-admin-node-centos-6-4 mycluster]$ ceph -bash: ceph: command not found But now, I go to the MON (I have only one) node and issue ceph -w or ceph health it throws errors: [ceph@ceph-node1-mon-centos-6-4 ceph]$ ceph -w 2013-10-31 14:52:58.601016 7fcd19b11700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-10-31 14:52:58.601063 7fcd19b11700 0 librados: client.admin initialization error (2) No such file or directory Error connecting to cluster: ObjectNotFound [ceph@ceph-node1-mon-centos-6-4 ceph]$ ceph health 2013-10-31 14:53:03.675425 7fe4937fc700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-10-31 14:53:03.675467 7fe4937fc700 0 librados: client.admin initialization error (2) No such file or directory Error connecting to cluster: ObjectNotFound How do I know I am done installing ceph... finally? Shall I ignore this and install a ceph client vm to test object and block?? Thanks! Narendra -----Original Message----- From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx] Sent: Thursday, October 31, 2013 11:04 AM To: Trivedi, Narendra Cc: Karan Singh; alistair whittle; ceph-users@xxxxxxxxxxxxxx Subject: Re: Activating OSDs takes forever On Thu, Oct 31, 2013 at 12:01 PM, Trivedi, Narendra <Narendra.Trivedi@xxxxxxxxxx> wrote: > After activating OSDs, I tried to use ceph-deploy to copy the > configuration file and admin ket to my admin node and my ceph nodes so > that I can use ceph CLI without having to specify the monitor address > and ceph.client.admin.keyring each time I execute a command but I got > the following error: > > > > [ceph@ceph-admin-node-centos-6-4 mycluster]$ ceph-deploy admin > ceph-admin-node-centos-6-4 ceph-node1-mon-centos-6-4 > ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy admin > ceph-admin-node-centos-6-4 ceph-node1-mon-centos-6-4 > ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 > > [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to > ceph-admin-node-centos-6-4 > > [ceph_deploy.sudo_pushy][DEBUG ] will use a local connection with sudo > > Traceback (most recent call last): > > File "/usr/bin/ceph-deploy", line 21, in <module> > > sys.exit(main()) > > File > "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", > line 83, in newfunc > > return f(*a, **kw) > > File "/usr/lib/python2.6/site-packages/ceph_deploy/cli.py", line > 150, in main > > return args.func(args) > > File "/usr/lib/python2.6/site-packages/ceph_deploy/admin.py", line > 40, in admin > > overwrite=args.overwrite_conf, > > File "/usr/lib/python2.6/site-packages/pushy/protocol/proxy.py", > line 255, in <lambda> > > (conn.operator(type_, self, args, kwargs)) > > File > "/usr/lib/python2.6/site-packages/pushy/protocol/connection.py", line > 66, in operator > > return self.send_request(type_, (object, args, kwargs)) > > File > "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", > line 329, in send_request > > return self.__handle(m) > > File > "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", > line 645, in __handle > > raise e > > pushy.protocol.proxy.ExceptionProxy: [Errno 2] No such file or directory: > '/etc/ceph/ceph.conf.16463.tmp' > > > > Has anyone seen anything like this? Yes, this should be fixed in the upcoming ceph-deploy version. It happens when you attempt to run ceph-deploy on a node that doesn't have ceph installed. This happens because the `/etc/ceph` directory doesn't exist, which is created when ceph is installed. If ceph was installed and this is still happening is because purge or purgedata removed that directory. The current workaround is to `mkdir /etc/ceph` if ceph is in fact installed. > > > > > > > > From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx] > Sent: Thursday, October 31, 2013 8:19 AM > To: Karan Singh > Cc: alistair whittle; Trivedi, Narendra; ceph-users@xxxxxxxxxxxxxx > > > Subject: Re: Activating OSDs takes forever > > > > > > > > On Thu, Oct 31, 2013 at 8:07 AM, Karan Singh <ksingh@xxxxxx> wrote: > > You are right AW , i also faced this very same error message apparently > everything just works fine till now no issues reported yet. > > > > Regards > > Karan Singh > > > > ________________________________ > > From: "alistair whittle" <alistair.whittle@xxxxxxxxxxxx> > To: "Narendra Trivedi" <Narendra.Trivedi@xxxxxxxxxx>, > ceph-users@xxxxxxxxxxxxxx > Sent: Thursday, 31 October, 2013 12:06:52 PM > Subject: Re: Activating OSDs takes forever > > > > > > I got these exact same error messages on my RHEL build. It did however > seem that my OSD's were correctly built and brought online. Never got to > the bottom of these errors though. > > > > There were others on this forum that had similar issues, and theirs > too seemed to be working successfully despite the apparent problems > during activation. > > > > From: ceph-users-bounces@xxxxxxxxxxxxxx > [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Trivedi, > Narendra > Sent: Wednesday, October 30, 2013 8:58 PM > To: ceph-users@xxxxxxxxxxxxxx > Subject: Activating OSDs takes forever > > > > Hi All, > > > > I had a pretty good run until I issued a command to activate OSDs. Now > I am back with some more problemsL. My setup is exactly like the one > in the official ceph documentation: > > > > http://ceph.com/docs/master/start/quick-ceph-deploy/ > > > > That means, I am just using node2:/tmp/osd0 and node3:/tmp/osd1 as > OSDs. I am running 64-bit CentOS-6-4. > > > > I was able to prepare OSDs fine but when I am activating them it just > takes forver (15 mins and counting): > > > > [ceph@ceph-admin-node-centos-6-4 mycluster]$ ceph-deploy osd activate > ceph-node2-osd0-centos-6-4:/tmp/osd0 > ceph-node3-osd1-centos-6-4:/tmp/osd1 > > [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy osd > activate > ceph-node2-osd0-centos-6-4:/tmp/osd0 > ceph-node3-osd1-centos-6-4:/tmp/osd1 > > [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks > ceph-node2-osd0-centos-6-4:/tmp/osd0: ceph-node3-osd1-centos-6-4:/tmp/osd1: > > [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with > sudo > > [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final > > [ceph_deploy.osd][DEBUG ] activating host ceph-node2-osd0-centos-6-4 > disk > /tmp/osd0 > > [ceph_deploy.osd][DEBUG ] will use init type: sysvinit > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: > ceph-disk-activate --mark-init sysvinit --mount /tmp/osd0 > > [ceph-node2-osd0-centos-6-4][INFO ] === osd.0 === > > [ceph-node2-osd0-centos-6-4][INFO ] Starting Ceph osd.0 on > ceph-node2-osd0-centos-6-4... > > [ceph-node2-osd0-centos-6-4][INFO ] starting osd.0 at :/0 osd_data > /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal > > [ceph-node2-osd0-centos-6-4][ERROR ] got latest monmap > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.059954 > 7faffbd537a0 > -1 journal FileJournal::_open: disabling aio for non-block journal. > Use journal_force_aio to force use of aio anyway > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.121705 > 7faffbd537a0 > -1 journal FileJournal::_open: disabling aio for non-block journal. > Use journal_force_aio to force use of aio anyway > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.122260 > 7faffbd537a0 > -1 filestore(/tmp/osd0) could not find 23c2fcde/osd_superblock/0//-1 > in > index: (2) No such file or directory > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.204635 > 7faffbd537a0 > -1 created object store /tmp/osd0 journal /tmp/osd0/journal for osd.0 > fsid ec8c48d5-3889-433b-bf68-558f2eb39a8c > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.204714 > 7faffbd537a0 > -1 auth: error reading file: /tmp/osd0/keyring: can't open > /tmp/osd0/keyring: (2) No such file or directory > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.204938 > 7faffbd537a0 > -1 created new key in keyring /tmp/osd0/keyring > > [ceph-node2-osd0-centos-6-4][ERROR ] added key for osd.0 > > [ceph-node2-osd0-centos-6-4][ERROR ] create-or-move updating item name > 'osd.0' weight 0 at location > {host=ceph-node2-osd0-centos-6-4,root=default} > to crush map > > > > Why is ceph-deploy throwing bunch of errors when all I did was to > create directories under /tmp on both nodes and issue a command to > prepare them without any problems. I mean I can always delete the > /tmp/osd0-1 and re-issue the commands again but I am wondering if > anyone else has been this before? > > > > > > The logging here is misleading because what is happening is that > ceph-deploy interprets "remote stderr output" as ERROR logging and > "remote stdout output" as DEBUG. So if the remote host is sending > output to `stderr` it will show as ERROR in ceph-deploy's logging. > > Unfortunately there are tools (and libraries!) that will default to > push all logging to `stderr`. There is little ceph-deploy can do here > because it can't interpret if the output is _really_ an error or is it > just logging something that is not really an error but somehow it is > going to `stderr`. > > I will make sure to add a not in the ceph-deploy docs about this so it > is a bit more clear. > > > > Thanks a lot in advance! > > Narendra > > > > > > > This message contains information which may be confidential and/or > privileged. Unless you are the intended recipient (or authorized to > receive for the intended recipient), you may not read, use, copy or > disclose to anyone the message or any information contained in the > message. If you have received the message in error, please advise the > sender by reply e-mail and delete the message and any attachment(s) > thereto without retaining any copies. > > _______________________________________________ > > This message is for information purposes only, it is not a > recommendation, advice, offer or solicitation to buy or sell a product > or service nor an official confirmation of any transaction. It is > directed at persons who are professionals and is not intended for > retail customer use. Intended for recipient only. This message is subject to the terms at: > www.barclays.com/emaildisclaimer. > > For important disclosures, please see: > www.barclays.com/salesandtradingdisclaimer regarding market commentary > from Barclays Sales and/or Trading, who are active market > participants; and in respect of Barclays Research, including > disclosures relating to specific issuers, please see http://publicresearch.barclays.com. > > _______________________________________________ > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > This message contains information which may be confidential and/or > privileged. Unless you are the intended recipient (or authorized to > receive for the intended recipient), you may not read, use, copy or > disclose to anyone the message or any information contained in the > message. If you have received the message in error, please advise the > sender by reply e-mail and delete the message and any attachment(s) > thereto without retaining any copies. This message contains information which may be confidential and/or privileged. Unless you are the intended recipient (or authorized to receive for the intended recipient), you may not read, use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail and delete the message and any attachment(s) thereto without retaining any copies. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |