On Sat, 2 Nov 2013, Trivedi, Narendra wrote: > > Hi Sage, > > I believe I issued a "ceph-deploy install..." from the admin node as per the > documentation and that was almost ok as per the output of the command below > except sometimes there?s an error followed by an ?OK? message (see the > highlighted item in the red below). I eventually ran into some permission > issues but seems things went okay: Hmm, the below output makes it look like it was successfully installed on node1 node2 and node3. Can you confirm that /etc/ceph exists on all three of those hosts? Oh, looking back at your original message, it looks like you are trying to create OSDs on /tmp/osd*. I would create directories like /ceph/osdo, /ceph/osd1, or similar. I believe you need to create the directories beforehand, too. (In a normal deployment, you are either feeding ceph raw disks (/dev/XXX) or an existing mount point on a dedicated disk you already configured and mounted.) sage > > > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy install > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy install > ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster > ceph hosts ceph-node1-mon-centos-6-4 ceph-node2-osd0-centos-6-4 > ceph-node3-osd1-centos-6-4 > > [ceph_deploy.install][DEBUG ] Detecting platform for host > ceph-node1-mon-centos-6-4 ... > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node1-mon-centos-6-4][INFO ] installing ceph on > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][INFO ] adding EPEL repository > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo wget > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node1-mon-centos-6-4][ERROR ] --2013-11-01 19:51:20-- > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node1-mon-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... > connected. > > [ceph-node1-mon-centos-6-4][ERROR ] Proxy request sent, awaiting response... > 200 OK > > [ceph-node1-mon-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] > > [ceph-node1-mon-centos-6-4][ERROR ] Saving to: > `epel-release-6-8.noarch.rpm.2' > > [ceph-node1-mon-centos-6-4][ERROR ] > > [ceph-node1-mon-centos-6-4][ERROR ] 0K .......... > .... 100% 4.79M=0.003s > > [ceph-node1-mon-centos-6-4][ERROR ] > > [ceph-node1-mon-centos-6-4][ERROR ] Last-modified header invalid -- > time-stamp ignored. > > [ceph-node1-mon-centos-6-4][ERROR ] 2013-11-01 19:52:20 (4.79 MB/s) - > `epel-release-6-8.noarch.rpm.2' saved [14540/14540] > > [ceph-node1-mon-centos-6-4][ERROR ] > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs epel-release-6*.rpm > > [ceph-node1-mon-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node1-mon-centos-6-4][DEBUG ] epel-release > ################################################## > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm --import > https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node1-mon-centos-6-4][DEBUG ] Retrieving > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node1-mon-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-release > ################################################## > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo yum -y -q install > ceph > > [ceph-node1-mon-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 already > installed and latest version > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph --version > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph version 0.67.4 > (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) > > [ceph_deploy.install][DEBUG ] Detecting platform for host > ceph-node2-osd0-centos-6-4 ... > > [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node2-osd0-centos-6-4][INFO ] installing ceph on > ceph-node2-osd0-centos-6-4 > > [ceph-node2-osd0-centos-6-4][INFO ] adding EPEL repository > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo wget > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node2-osd0-centos-6-4][ERROR ] --2013-11-01 19:52:54-- > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node2-osd0-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... > connected. > > [ceph-node2-osd0-centos-6-4][ERROR ] Proxy request sent, awaiting > response... 200 OK > > [ceph-node2-osd0-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] > > [ceph-node2-osd0-centos-6-4][ERROR ] Saving to: > `epel-release-6-8.noarch.rpm.1' > > [ceph-node2-osd0-centos-6-4][ERROR ] > > [ceph-node2-osd0-centos-6-4][ERROR ] 0K .......... > .... 100% 5.56M=0.002s > > [ceph-node2-osd0-centos-6-4][ERROR ] > > [ceph-node2-osd0-centos-6-4][ERROR ] Last-modified header invalid -- > time-stamp ignored. > > [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-01 19:53:54 (5.56 MB/s) - > `epel-release-6-8.noarch.rpm.1' saved [14540/14540] > > [ceph-node2-osd0-centos-6-4][ERROR ] > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs epel-release-6*.rpm > > [ceph-node2-osd0-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node2-osd0-centos-6-4][DEBUG ] epel-release > ################################################## > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm --import > https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node2-osd0-centos-6-4][DEBUG ] Retrieving > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node2-osd0-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node2-osd0-centos-6-4][DEBUG ] ceph-release > ################################################## > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo yum -y -q install > ceph > > [ceph-node2-osd0-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 > already installed and latest version > > [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo ceph --version > > [ceph-node2-osd0-centos-6-4][DEBUG ] ceph version 0.67.4 > (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) > > [ceph_deploy.install][DEBUG ] Detecting platform for host > ceph-node3-osd1-centos-6-4 ... > > [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final > > [ceph-node3-osd1-centos-6-4][INFO ] installing ceph on > ceph-node3-osd1-centos-6-4 > > [ceph-node3-osd1-centos-6-4][INFO ] adding EPEL repository > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo wget > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node3-osd1-centos-6-4][ERROR ] --2013-11-01 19:54:17-- > http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm > > [ceph-node3-osd1-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... > connected. > > [ceph-node3-osd1-centos-6-4][ERROR ] Proxy request sent, awaiting > response... 200 OK > > [ceph-node3-osd1-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] > > [ceph-node3-osd1-centos-6-4][ERROR ] Saving to: > `epel-release-6-8.noarch.rpm.1' > > [ceph-node3-osd1-centos-6-4][ERROR ] > > [ceph-node3-osd1-centos-6-4][ERROR ] 0K .......... > .... 100% 5.37M=0.003s > > [ceph-node3-osd1-centos-6-4][ERROR ] > > [ceph-node3-osd1-centos-6-4][ERROR ] Last-modified header invalid -- > time-stamp ignored. > > [ceph-node3-osd1-centos-6-4][ERROR ] 2013-11-01 19:55:16 (5.37 MB/s) - > `epel-release-6-8.noarch.rpm.1' saved [14540/14540] > > [ceph-node3-osd1-centos-6-4][ERROR ] > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs epel-release-6*.rpm > > [ceph-node3-osd1-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node3-osd1-centos-6-4][DEBUG ] epel-release > ################################################## > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm --import > https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm -Uvh > --replacepkgs > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node3-osd1-centos-6-4][DEBUG ] Retrieving > http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm > > [ceph-node3-osd1-centos-6-4][DEBUG ] Preparing... > ################################################## > > [ceph-node3-osd1-centos-6-4][DEBUG ] ceph-release > ################################################## > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo yum -y -q install > ceph > > [ceph-node3-osd1-centos-6-4][DEBUG ] Package ceph-0.67.4-0.el6.x86_64 > already installed and latest version > > [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo ceph --version > > [ceph-node3-osd1-centos-6-4][DEBUG ] ceph version 0.67.4 > (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy mon create > ceph-node1-mon-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy mon create > ceph-node1-mon-centos-6-4 > > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts > ceph-node1-mon-centos-6-4 > > [ceph_deploy.mon][DEBUG ] detecting platform for host > ceph-node1-mon-centos-6-4 ... > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.mon][INFO ] distro info: CentOS 6.4 Final > > [ceph-node1-mon-centos-6-4][DEBUG ] determining if provided host has same > hostname in remote > > [ceph-node1-mon-centos-6-4][DEBUG ] get remote short hostname > > [ceph-node1-mon-centos-6-4][DEBUG ] deploying mon to > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] get remote short hostname > > [ceph-node1-mon-centos-6-4][DEBUG ] remote hostname: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] write cluster configuration to > /etc/ceph/{cluster}.conf > > [ceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf > exists with different content; use --overwrite-conf to overwrite > > [ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors > > [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy mon create > ceph-node1-mon-centos-6-4 > > [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy mon create > ceph-node1-mon-centos-6-4 > > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts > ceph-node1-mon-centos-6-4 > > [ceph_deploy.mon][DEBUG ] detecting platform for host > ceph-node1-mon-centos-6-4 ... > > [ceph-node1-mon-centos-6-4][DEBUG ] connected to host: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] detect platform information from remote > host > > [ceph-node1-mon-centos-6-4][DEBUG ] detect machine type > > [ceph_deploy.mon][INFO ] distro info: CentOS 6.4 Final > > [ceph-node1-mon-centos-6-4][DEBUG ] determining if provided host has same > hostname in remote > > [ceph-node1-mon-centos-6-4][DEBUG ] get remote short hostname > > [ceph-node1-mon-centos-6-4][DEBUG ] deploying mon to > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] get remote short hostname > > [ceph-node1-mon-centos-6-4][DEBUG ] remote hostname: > ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] write cluster configuration to > /etc/ceph/{cluster}.conf > > [ceph-node1-mon-centos-6-4][DEBUG ] create the mon path if it does not exist > > [ceph-node1-mon-centos-6-4][DEBUG ] checking for done path: > /var/lib/ceph/mon/ceph-ceph-node1-mon-centos-6-4/done > > [ceph-node1-mon-centos-6-4][DEBUG ] done path does not exist: > /var/lib/ceph/mon/ceph-ceph-node1-mon-centos-6-4/done > > [ceph-node1-mon-centos-6-4][INFO ] creating tmp path: /var/lib/ceph/tmp > > [ceph-node1-mon-centos-6-4][INFO ] creating keyring file: > /var/lib/ceph/tmp/ceph-ceph-node1-mon-centos-6-4.mon.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] create the monitor keyring file > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph-mon --cluster > ceph --mkfs -i ceph-node1-mon-centos-6-4 --keyring > /var/lib/ceph/tmp/ceph-ceph-node1-mon-centos-6-4.mon.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-mon: mon.noname-a 10.12.0.70:6789/0 > is local, renaming to mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-mon: set fsid to > c732fc5f-a656-401a-a8e5-4bfed1f89d20 > > [ceph-node1-mon-centos-6-4][DEBUG ] ceph-mon: created monfs at > /var/lib/ceph/mon/ceph-ceph-node1-mon-centos-6-4 for > mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][INFO ] unlinking keyring file > /var/lib/ceph/tmp/ceph-ceph-node1-mon-centos-6-4.mon.keyring > > [ceph-node1-mon-centos-6-4][DEBUG ] create a done file to avoid re-doing the > mon deployment > > [ceph-node1-mon-centos-6-4][DEBUG ] create the init path if it does not > exist > > [ceph-node1-mon-centos-6-4][DEBUG ] locating the `service` executable... > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo /sbin/service ceph > -c /etc/ceph/ceph.conf start mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] === mon.ceph-node1-mon-centos-6-4 === > > [ceph-node1-mon-centos-6-4][DEBUG ] Starting Ceph > mon.ceph-node1-mon-centos-6-4 on ceph-node1-mon-centos-6-4... > > [ceph-node1-mon-centos-6-4][DEBUG ] Starting ceph-create-keys on > ceph-node1-mon-centos-6-4... > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph > --cluster=ceph --admin-daemon > /var/run/ceph/ceph-mon.ceph-node1-mon-centos-6-4.asok mon_status > > [ceph-node1-mon-centos-6-4][DEBUG ]*************************************************************************** > ***** > > [ceph-node1-mon-centos-6-4][DEBUG ] status for monitor: > mon.ceph-node1-mon-centos-6-4 > > [ceph-node1-mon-centos-6-4][DEBUG ] { > > [ceph-node1-mon-centos-6-4][DEBUG ] "election_epoch": 2, > > [ceph-node1-mon-centos-6-4][DEBUG ] "extra_probe_peers": [], > > [ceph-node1-mon-centos-6-4][DEBUG ] "monmap": { > > [ceph-node1-mon-centos-6-4][DEBUG ] "created": "0.000000", > > [ceph-node1-mon-centos-6-4][DEBUG ] "epoch": 1, > > [ceph-node1-mon-centos-6-4][DEBUG ] "fsid": > "c732fc5f-a656-401a-a8e5-4bfed1f89d20", > > [ceph-node1-mon-centos-6-4][DEBUG ] "modified": "0.000000", > > [ceph-node1-mon-centos-6-4][DEBUG ] "mons": [ > > [ceph-node1-mon-centos-6-4][DEBUG ] { > > [ceph-node1-mon-centos-6-4][DEBUG ] "addr": "10.12.0.70:6789/0", > > [ceph-node1-mon-centos-6-4][DEBUG ] "name": > "ceph-node1-mon-centos-6-4", > > [ceph-node1-mon-centos-6-4][DEBUG ] "rank": 0 > > [ceph-node1-mon-centos-6-4][DEBUG ] } > > [ceph-node1-mon-centos-6-4][DEBUG ] ] > > [ceph-node1-mon-centos-6-4][DEBUG ] }, > > [ceph-node1-mon-centos-6-4][DEBUG ] "name": "ceph-node1-mon-centos-6-4", > > [ceph-node1-mon-centos-6-4][DEBUG ] "outside_quorum": [], > > [ceph-node1-mon-centos-6-4][DEBUG ] "quorum": [ > > [ceph-node1-mon-centos-6-4][DEBUG ] 0 > > [ceph-node1-mon-centos-6-4][DEBUG ] ], > > [ceph-node1-mon-centos-6-4][DEBUG ] "rank": 0, > > [ceph-node1-mon-centos-6-4][DEBUG ] "state": "leader", > > [ceph-node1-mon-centos-6-4][DEBUG ] "sync_provider": [] > > [ceph-node1-mon-centos-6-4][DEBUG ] } > > [ceph-node1-mon-centos-6-4][DEBUG ]*************************************************************************** > ***** > > [ceph-node1-mon-centos-6-4][INFO ] monitor: mon.ceph-node1-mon-centos-6-4 > is running > > [ceph-node1-mon-centos-6-4][INFO ] Running command: sudo ceph > --cluster=ceph --admin-daemon > /var/run/ceph/ceph-mon.ceph-node1-mon-centos-6-4.asok mon_status > > > > Thanks! > > Narendra > > -----Original Message----- > From: Sage Weil [mailto:sage@xxxxxxxxxxx] > Sent: Friday, November 01, 2013 8:37 PM > To: Trivedi, Narendra > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Very frustrated with Ceph! > > > > On Sat, 2 Nov 2013, Trivedi, Narendra wrote: > > > [ceph-node2-osd0-centos-6-4][WARNIN] osd keyring does not exist yet, > > > creating one > > > > > > [ceph-node2-osd0-centos-6-4][DEBUG ] create a keyring file > > > > > > [ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory > > > > Did you do 'ceph-deploy install ...' on these hosts? > > > > sage > > > This message contains information which may be confidential and/or > privileged. Unless you are the intended recipient (or authorized to receive > for the intended recipient), you may not read, use, copy or disclose to > anyone the message or any information contained in the message. If you have > received the message in error, please advise the sender by reply e-mail and > delete the message and any attachment(s) thereto without retaining any > copies. > >
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
