On Mon, Nov 4, 2013 at 12:25 PM, Gruher, Joseph R <joseph.r.gruher@xxxxxxxxx> wrote: > Could these problems be caused by running a purgedata but not a purge? It could be, I am not clear on what the expectation was for just doing purgedata without a purge. > Purgedata removes /etc/ceph but without the purge ceph is still installed, > then ceph-deploy install detects ceph as already installed and does not > (re)create /etc/ceph? ceph-deploy will not create directories for you, that is left to the ceph install process, and just to be clear, the latest ceph-deploy version (1.3) does not remote /etc/ceph, just the contents. > > > > [ceph-node2-osd0-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 > already installed and latest version > > > > I wonder if you ran a purge and a purgedata if you might have better luck. > That always works for me. > > > > From: ceph-users-bounces@xxxxxxxxxxxxxx > [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Trivedi, Narendra > Sent: Saturday, November 02, 2013 10:42 PM > To: Sage Weil > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Very frustrated with Ceph! > > > > Thanks a lot Sage for your help :-). > > > > I started from scratch: See the commands and output below: > > > > 1) First of all, all the nodes did have but in order to start from scratch I > removed /etc/ceph from each node. > > > > 2) I issued a ceph-deploy purgedata to each node from the admin node. This > threw error towards the end. I assuming since I manually removed /etc/ceph > from nodes and hence rm command fails: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy purgedata > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy purgedata > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.install][DEBUG ] Purging data from cluster ceph hosts > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo which ceph > > [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo which ceph > > [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo which ceph > > ceph is still installed on: ['ceph-node1-mon-centos-6-4', > 'ceph-node2-osd0-centos-6-4', 'ceph-node3-osd1-centos-6-4'] > > Continue (y/n)y > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node1-mon-centos-6-4][INFO ] purging data on > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /var/lib/ceph > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /etc/ceph/* > > [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node2-osd0-centos-6-4][INFO ] purging data on > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /var/lib/ceph > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /etc/ceph/* > > Exception in thread Thread-1 (most likely raised during interpreter > shutdown): > > Traceback (most recent call last): > > File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner > > File "", line 89, in run > > <type 'exceptions.KeyError'>: <WorkerThread(Thread-1, started daemon > 140730692818688)> > > [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node3-osd1-centos-6-4][INFO ] purging data on > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /var/lib/ceph > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rm -rf > --one-file-system -- /etc/ceph/* > > Exception in thread Thread-1 (most likely raised during interpreter > shutdown): > > Traceback (most recent call last): > > File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner > > File "", line 89, in run > > <type 'exceptions.KeyError'>: <WorkerThread(Thread-1, started daemon > 140094013404928)> > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy forgetkeys > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy forgetkeys > > > > 3) On the admin node, clean up and create a new directory: > > > > [ceph@ceph-admin-node-centos-6-4 ~]$ \rm -rf my-cluster/ > > [ceph@ceph-admin-node-centos-6-4 ~]$ mkdir my-cluster > > [ceph@ceph-admin-node-centos-6-4 ~]$ cd my-cluster > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ > > > > 4) Create /etc/ceph on all the nodes and /ceph/osd0 and /ceph/osd1 on nodes > 2 and 3: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ssh > ceph@ceph-node1-mon-centos-6-4 > > Last login: Sat Nov 2 23:02:29 2013 from 10.12.132.70 > > [ceph@ceph-node1-mon-centos-6-4 ~]$ ls /etc/ceph > > ls: cannot access /etc/ceph: No such file or directory > > [ceph@ceph-node1-mon-centos-6-4 ~]$ sudo mkdir /etc/ceph > > [ceph@ceph-node1-mon-centos-6-4 ~]$ exit > > logout > > Connection to ceph-node1-mon-centos-6-4 closed. > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ssh > ceph@ceph-node2-osd0-centos-6-4 > > Last login: Sat Nov 2 23:02:41 2013 from 10.12.132.70 > > [ceph@ceph-node2-osd0-centos-6-4 ~]$ ls /etc/ceph > > ls: cannot access /etc/ceph: No such file or directory > > [ceph@ceph-node2-osd0-centos-6-4 ~]$ sudo mkdir /etc/ceph > > [ceph@ceph-node2-osd0-centos-6-4 /]$ ls /ceph > > ls: cannot access /ceph: No such file or directory > > [ceph@ceph-node2-osd0-centos-6-4 /]$ sudo mkdir -p /ceph/osd0 > > [ceph@ceph-node2-osd0-centos-6-4 /]$ exit > > logout > > Connection to ceph-node2-osd0-centos-6-4 closed. > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ssh > ceph@ceph-node3-osd1-centos-6-4 > > Last login: Sat Nov 2 23:02:49 2013 from 10.12.132.70 > > [ceph@ceph-node3-osd1-centos-6-4 ~]$ ls /etc/ceph > > ls: cannot access /etc/ceph: No such file or directory > > [ceph@ceph-node3-osd1-centos-6-4 ~]$ sudo mkdir /etc/ceph > > [ceph@ceph-node3-osd1-centos-6-4 ~]$ ls /ceph > > ls: cannot access /ceph: No such file or directory > > [ceph@ceph-node3-osd1-centos-6-4 ~]$ sudo mkdir -p /ceph/osd1 > > [ceph@ceph-node3-osd1-centos-6-4 ~]$ exit > > logout > > Connection to ceph-node3-osd1-centos-6-4 closed. > > > > 5) Create a new cluster: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ls > > ceph.log > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy new > ceph-node1-mon-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy new > ceph-node1-mon-centos-6-4 > > [ceph_deploy.new][DEBUG ] Creating new cluster named ceph > > [ceph_deploy.new][DEBUG ] Resolving host ceph-node1-mon-centos-6-4 > > [ceph_deploy.new][DEBUG ] Monitor ceph-node1-mon-centos-6-4 at 10.12.0.70 > > [ceph_deploy.new][DEBUG ] Monitor initial members are > ['ceph-node1-mon-centos-6-4'] > > [ceph_deploy.new][DEBUG ] Monitor addrs are ['10.12.0.70'] > > [ceph_deploy.new][DEBUG ] Creating a random mon key... > > [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf... > > [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ls > > ceph.conf ceph.log ceph.mon.keyring > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ cat ceph.conf > > [global] > > filestore_xattr_use_omap = true > > mon_host = 10.12.0.70 > > fsid = 4c34c059-41a8-4820-bfb3-b9bd480267e8 > > mon_initial_members = ceph-node1-mon-centos-6-4 > > auth_supported = cephx > > osd_journal_size = 1024 > > > > 6) Install Ceph: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy install > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy install > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster > ceph hosts ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.install][DEBUG ] Detecting platform for host > ceph-node1-mon-centos-6-4 ... > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node1-mon-centos-6-4][INFO ] installing ceph on > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][INFO ] adding EPEL repository > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo wget > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node1-mon-centos-6-4][ERROR ] --2013-11-03 00:03:02-- > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node1-mon-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... > connected. > > [ceph-node1-mon-centos-6-4][ERROR ] Proxy request sent, awaiting response... > 200 OK > > [ceph-node1-mon-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] > > [ceph-node1-mon-centos-6-4][ERROR ] Saving to: > `epel-release-6-8.noarch.rpm.5' > > [ceph-node1-mon-centos-6-4][ERROR ] > > [ceph-node1-mon-centos-6-4][ERROR ] 0K .......... .... > 100% 401K=0.04s > > [ceph-node1-mon-centos-6-4][ERROR ] > > [ceph-node1-mon-centos-6-4][ERROR ] Last-modified header invalid -- > time-stamp ignored. > > [ceph-node1-mon-centos-6-4][ERROR ] 2013-11-03 00:04:02 (401 KB/s) - > `epel-release-6-8.noarch.rpm.5' saved [14540/14540] > > [ceph-node1-mon-centos-6-4][ERROR ] > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs epel-release-6*.rpm > > [ceph-node1-mon-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node1-mon-centos-6-4][DEBUG ] epel-release > ################################################## > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm --import > https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node1-mon-centos-6-4][DEBUG ] Retrieving > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node1-mon-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-release > ################################################## > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo yum -y -q install > ceph > > [ceph-node1-mon-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 already > installed and latest version > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph --version > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph version 0.67.4 > (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) > > [ceph_deploy.install][DEBUG ] Detecting platform for host > ceph-node2-osd0-centos-6-4 ... > > [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node2-osd0-centos-6-4][INFO ] installing ceph on > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][INFO ] adding EPEL repository > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo wget > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node2-osd0-centos-6-4][ERROR ] --2013-11-03 00:04:43-- > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node2-osd0-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... > connected. > > [ceph-node2-osd0-centos-6-4][ERROR ] Proxy request sent, awaiting > response... 200 OK > > [ceph-node2-osd0-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] > > [ceph-node2-osd0-centos-6-4][ERROR ] Saving to: > `epel-release-6-8.noarch.rpm.4' > > [ceph-node2-osd0-centos-6-4][ERROR ] > > [ceph-node2-osd0-centos-6-4][ERROR ] 0K .......... .... > 100% 412K=0.03s > > [ceph-node2-osd0-centos-6-4][ERROR ] > > [ceph-node2-osd0-centos-6-4][ERROR ] Last-modified header invalid -- > time-stamp ignored. > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-03 00:05:43 (412 KB/s) - > `epel-release-6-8.noarch.rpm.4' saved [14540/14540] > > [ceph-node2-osd0-centos-6-4][ERROR ] > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs epel-release-6*.rpm > > [ceph-node2-osd0-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node2-osd0-centos-6-4][DEBUG ] epel-release > ################################################## > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm --import > https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node2-osd0-centos-6-4][DEBUG ] Retrieving > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node2-osd0-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node2-osd0-centos-6-4][DEBUG ] ceph-release > ################################################## > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo yum -y -q install > ceph > > [ceph-node2-osd0-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 > already installed and latest version > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo ceph --version > > [ceph-node2-osd0-centos-6-4][DEBUG ] ceph version 0.67.4 > (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) > > [ceph_deploy.install][DEBUG ] Detecting platform for host > ceph-node3-osd1-centos-6-4 ... > > [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node3-osd1-centos-6-4][INFO ] installing ceph on > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][INFO ] adding EPEL repository > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo wget > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node3-osd1-centos-6-4][ERROR ] --2013-11-03 00:06:01-- > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node3-osd1-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... > connected. > > [ceph-node3-osd1-centos-6-4][ERROR ] Proxy request sent, awaiting > response... 200 OK > > [ceph-node3-osd1-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] > > [ceph-node3-osd1-centos-6-4][ERROR ] Saving to: > `epel-release-6-8.noarch.rpm.4' > > [ceph-node3-osd1-centos-6-4][ERROR ] > > [ceph-node3-osd1-centos-6-4][ERROR ] 0K .......... .... > 100% 428K=0.03s > > [ceph-node3-osd1-centos-6-4][ERROR ] > > [ceph-node3-osd1-centos-6-4][ERROR ] Last-modified header invalid -- > time-stamp ignored. > > [ceph-node3-osd1-centos-6-4][ERROR ] 2013-11-03 00:07:01 (428 KB/s) - > `epel-release-6-8.noarch.rpm.4' saved [14540/14540] > > [ceph-node3-osd1-centos-6-4][ERROR ] > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs epel-release-6*.rpm > > [ceph-node3-osd1-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node3-osd1-centos-6-4][DEBUG ] epel-release > ################################################## > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm --import > https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node3-osd1-centos-6-4][DEBUG ] Retrieving > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node3-osd1-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node3-osd1-centos-6-4][DEBUG ] ceph-release > ################################################## > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo yum -y -q install > ceph > > [ceph-node3-osd1-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 > already installed and latest version > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo ceph --version > > [ceph-node3-osd1-centos-6-4][DEBUG ] ceph version 0.67.4 > (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ > > > > 7) Add a Ceph Monitor: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy mon create > ceph-node1-mon-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy mon create > ceph-node1-mon-centos-6-4 > > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts > ceph-node1-mon-centos-6-4 > > [ceph_deploy.mon][DEBUG ] detecting platform for host > ceph-node1-mon-centos-6-4 ... > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.mon][INFO ] distro info: CentOS 6.4 Final > > [ceph-node1-mon-centos-6-4][DEBUG ] determining if provided host has same > hostname in remote > > [ceph-node1-mon-centos-6-4][DEBUG ] get remote short hostname > > [ceph-node1-mon-centos-6-4][DEBUG ] deploying mon to > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] get remote short hostname > > [ceph-node1-mon-centos-6-4][DEBUG ] remote hostname: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] write cluster configuration to > /etc/ceph/{cluster}.conf > > [ceph-node1-mon-centos-6-4][DEBUG ] create the mon path if it does not exist > > [ceph-node1-mon-centos-6-4][DEBUG ] checking for done path: > /var/lib/ceph/mon/ceph-ceph-node1-mon-centos-6-4/done > > [ceph-node1-mon-centos-6-4][DEBUG ] done path does not exist: > /var/lib/ceph/mon/ceph-ceph-node1-mon-centos-6-4/done > > [ceph-node1-mon-centos-6-4][INFO ] creating tmp path: /var/lib/ceph/tmp > > [ceph-node1-mon-centos-6-4][INFO ] creating keyring file: > /var/lib/ceph/tmp/ceph-ceph-node1-mon-centos-6-4.mon.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] create the monitor keyring file > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph-mon --cluster > ceph --mkfs -i ceph-node1-mon-centos-6-4 --keyring > /var/lib/ceph/tmp/ceph-ceph-node1-mon-centos-6-4.mon.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-mon: mon.noname-a 10.12.0.70:6789/0 > is local, renaming to mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-mon: set fsid to > 4c34c059-41a8-4820-bfb3-b9bd480267e8 > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-mon: created monfs at > /var/lib/ceph/mon/ceph-ceph-node1-mon-centos-6-4 for > mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][INFO ] unlinking keyring file > /var/lib/ceph/tmp/ceph-ceph-node1-mon-centos-6-4.mon.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] create a done file to avoid re-doing the > mon deployment > > [ceph-node1-mon-centos-6-4][DEBUG ] create the init path if it does not > exist > > [ceph-node1-mon-centos-6-4][DEBUG ] locating the `service` executable... > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo /sbin/service ceph > -c /etc/ceph/ceph.conf start mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] === mon.ceph-node1-mon-centos-6-4 === > > [ceph-node1-mon-centos-6-4][DEBUG ] Starting Ceph > mon.ceph-node1-mon-centos-6-4 on ceph-node1-mon-centos-6-4... > > [ceph-node1-mon-centos-6-4][DEBUG ] Starting ceph-create-keys on > ceph-node1-mon-centos-6-4... > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph > --cluster=ceph --admin-daemon > /var/run/ceph/ceph-mon.ceph-node1-mon-centos-6-4.asok mon_status > > [ceph-node1-mon-centos-6-4][DEBUG ] > ******************************************************************************** > > [ceph-node1-mon-centos-6-4][DEBUG ] status for monitor: > mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] { > > [ceph-node1-mon-centos-6-4][DEBUG ] "election_epoch": 2, > > [ceph-node1-mon-centos-6-4][DEBUG ] "extra_probe_peers": [], > > [ceph-node1-mon-centos-6-4][DEBUG ] "monmap": { > > [ceph-node1-mon-centos-6-4][DEBUG ] "created": "0.000000", > > [ceph-node1-mon-centos-6-4][DEBUG ] "epoch": 1, > > [ceph-node1-mon-centos-6-4][DEBUG ] "fsid": > "4c34c059-41a8-4820-bfb3-b9bd480267e8", > > [ceph-node1-mon-centos-6-4][DEBUG ] "modified": "0.000000", > > [ceph-node1-mon-centos-6-4][DEBUG ] "mons": [ > > [ceph-node1-mon-centos-6-4][DEBUG ] { > > [ceph-node1-mon-centos-6-4][DEBUG ] "addr": "10.12.0.70:6789/0", > > [ceph-node1-mon-centos-6-4][DEBUG ] "name": > "ceph-node1-mon-centos-6-4", > > [ceph-node1-mon-centos-6-4][DEBUG ] "rank": 0 > > [ceph-node1-mon-centos-6-4][DEBUG ] } > > [ceph-node1-mon-centos-6-4][DEBUG ] ] > > [ceph-node1-mon-centos-6-4][DEBUG ] }, > > [ceph-node1-mon-centos-6-4][DEBUG ] "name": "ceph-node1-mon-centos-6-4", > > [ceph-node1-mon-centos-6-4][DEBUG ] "outside_quorum": [], > > [ceph-node1-mon-centos-6-4][DEBUG ] "quorum": [ > > [ceph-node1-mon-centos-6-4][DEBUG ] 0 > > [ceph-node1-mon-centos-6-4][DEBUG ] ], > > [ceph-node1-mon-centos-6-4][DEBUG ] "rank": 0, > > [ceph-node1-mon-centos-6-4][DEBUG ] "state": "leader", > > [ceph-node1-mon-centos-6-4][DEBUG ] "sync_provider": [] > > [ceph-node1-mon-centos-6-4][DEBUG ] } > > [ceph-node1-mon-centos-6-4][DEBUG ] > ******************************************************************************** > > [ceph-node1-mon-centos-6-4][INFO ] monitor: mon.ceph-node1-mon-centos-6-4 > is running > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph > --cluster=ceph --admin-daemon > /var/run/ceph/ceph-mon.ceph-node1-mon-centos-6-4.asok mon_status > > > > 8) Gather Keys: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ls > > ceph.conf ceph.log ceph.mon.keyring > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy gatherkeys > ceph-node1-mon-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy gatherkeys > ceph-node1-mon-centos-6-4 > > [ceph_deploy.gatherkeys][DEBUG ] Checking ceph-node1-mon-centos-6-4 for > /etc/ceph/ceph.client.admin.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph-node1-mon-centos-6-4][DEBUG ] fetch remote file > > [ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from > ceph-node1-mon-centos-6-4. > > [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring > > [ceph_deploy.gatherkeys][DEBUG ] Checking ceph-node1-mon-centos-6-4 for > /var/lib/ceph/bootstrap-osd/ceph.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph-node1-mon-centos-6-4][DEBUG ] fetch remote file > > [ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from > ceph-node1-mon-centos-6-4. > > [ceph_deploy.gatherkeys][DEBUG ] Checking ceph-node1-mon-centos-6-4 for > /var/lib/ceph/bootstrap-mds/ceph.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph-node1-mon-centos-6-4][DEBUG ] fetch remote file > > [ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from > ceph-node1-mon-centos-6-4. > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ls > > ceph.bootstrap-mds.keyring ceph.bootstrap-osd.keyring > ceph.client.admin.keyring ceph.conf ceph.log ceph.mon.keyring > > > > 9) Add two OSDs. Since I have already created /ceph/OSD0 and /ceph/OSD1 > (step 4 above), I am going to just issue a osd prepare command: > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy osd prepare > ceph-node2-osd0-centos-6-4:/ceph/osd0 ceph-node3-osd1-centos-6-4:/ceph/osd1 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy osd prepare > ceph-node2-osd0-centos-6-4:/ceph/osd0 ceph-node3-osd1-centos-6-4:/ceph/osd1 > > [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks > ceph-node2-osd0-centos-6-4:/ceph/osd0: > ceph-node3-osd1-centos-6-4:/ceph/osd1: > > [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final > > [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] write cluster configuration to > /etc/ceph/{cluster}.conf > > [ceph-node2-osd0-centos-6-4][WARNIN] osd keyring does not exist yet, > creating one > > [ceph-node2-osd0-centos-6-4][DEBUG ] create a keyring file > > [ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory > > [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final > > [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] write cluster configuration to > /etc/ceph/{cluster}.conf > > [ceph-node3-osd1-centos-6-4][WARNIN] osd keyring does not exist yet, > creating one > > [ceph-node3-osd1-centos-6-4][DEBUG ] create a keyring file > > [ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory > > [ceph_deploy][ERROR ] GenericError: Failed to create 2 OSDs > > > > Why is it failing now? > > > > Do you want me to re-create the VMs and re-issue everything? Please let me > know. > > > > Thanks a lot for your help! > > Narendra > > > > > > > > From: Sage Weil [mailto:sage@xxxxxxxxxxx] > Sent: Friday, November 01, 2013 10:13 PM > To: Trivedi, Narendra > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: RE: Very frustrated with Ceph! > > > > On Sat, 2 Nov 2013, Trivedi, Narendra wrote: > >> > >> Hi Sage, > >> > >> I believe I issued a "ceph-deploy install..." from the admin node as > >> per the documentation and that was almost ok as per the output of the > >> command below except sometimes there?s an error followed by an ?OK? > >> message (see the highlighted item in the red below). I eventually ran > >> into some permission issues but seems things went okay: > > > > Hmm, the below output makes it look like it was successfully installed on > > node1 node2 and node3. Can you confirm that /etc/ceph exists on all three > of those hosts? > > > > Oh, looking back at your original message, it looks like you are trying to > create OSDs on /tmp/osd*. I would create directories like /ceph/osdo, > /ceph/osd1, or similar. I believe you need to create the directories > beforehand, too. (In a normal deployment, you are either feeding ceph raw > disks (/dev/XXX) or an existing mount point on a dedicated disk you already > configured and mounted.) > > > > sage > > > > > > > > > > > > > This message contains information which may be confidential and/or > privileged. Unless you are the intended recipient (or authorized to receive > for the intended recipient), you may not read, use, copy or disclose to > anyone the message or any information contained in the message. If you have > received the message in error, please advise the sender by reply e-mail and > delete the message and any attachment(s) thereto without retaining any > copies. > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |