On Thu, Aug 14, 2014 at 2:28 AM, NotExist <notexist at gmail.com> wrote: > Hello everyone: > > Since there's no cuttlefish package for 14.04 server on ceph > repository (only ceph-deploy there), I tried to build cuttlefish from > source on 14.04. ...why? Cuttlefish is old and no longer provided updates. You really want to be using either Dumpling or Firefly. > > Here's what I did: > Get source by following http://ceph.com/docs/master/install/clone-source/ > Enter the sourcecode directory > git checkout cluttlefish > git submodule update > rm -rf src/civetweb/ src/erasure-code/ src/rocksdb/ > to get the latest cluttlefish repo. > > Build source by following http://ceph.com/docs/master/install/build-ceph/ > beside the package this url mentioned for Ubuntu: > > sudo apt-get install autotools-dev autoconf automake cdbs gcc g++ git > libboost-dev libedit-dev libssl-dev libtool libfcgi libfcgi-dev > libfuse-dev linux-kernel-headers libcrypto++-dev libcrypto++ > libexpat1-dev pkg-config > sudo apt-get install uuid-dev libkeyutils-dev libgoogle-perftools-dev > libatomic-ops-dev libaio-dev libgdata-common libgdata13 libsnappy-dev > libleveldb-dev > > I also found it will need > > sudo apt-get install libboost-filesystem-dev libboost-thread-dev > libboost-program-options-dev > > (And xfsprogs if you need xfs) > after all packages are installed, I start to complie according to the doc: > > ./autogen.sh > ./configure > make -j8 > > And install following > http://ceph.com/docs/master/install/install-storage-cluster/#installing-a-build > > sudo make install > > everything seems fine, but I found ceph_common.sh had been putted to > '/usr/local/lib/ceph', and some tools are putted into > /usr/local/usr/local/sbin/ (ceph-disk* and ceph-create-keys). I was > used to use ceph-disk to prepare the disk on other deployment (on > other machines with Emperor), but I can't do it now (and maybe the > path is the reason) so I choose to do do all stuffs manually. > > I follow the doc > http://ceph.com/docs/master/install/manual-deployment/ to deploy the > cluster many times, but it turns out different this time. > /etc/ceph isn't there, therefore I sudo mkdir /etc/ceph > Put a ceph.conf into /etc/ceph > Generate all required keys in /etc/ceph instead of /tmp/ to keep them > > ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n > mon. --cap mon 'allow *' > ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring > --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd > 'allow *' --cap mds 'allow' > ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring > /etc/ceph/ceph.client.admin.keyring > > Generate monmap with monmaptool > > monmaptool --create --add storage01 192.168.11.1 --fsid > 9f8fffe3-040d-4641-b35a-ffa90241f723 /etc/ceph/monmap > > /var/lib/ceph is not there either > > sudo mkdir -p /var/lib/ceph/mon/ceph-storage01 > sudo ceph-mon --mkfs -i storage01 --monmap /etc/ceph/monmap --keyring > /etc/ceph/ceph.mon.keyring > > log directory are not there, so I create it manually: > > sudo mkdir /var/log/ceph > > since service doesn't work, I start mon daemon manually: > > sudo /usr/local/bin/ceph-mon -i storage01 > > and ceph -s looks like these: > storage at storage01:~/ceph$ ceph -s > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds > monmap e1: 1 mons at {storage01=192.168.11.1:6789/0}, election > epoch 2, quorum 0 storage01 > osdmap e1: 0 osds: 0 up, 0 in > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail > mdsmap e1: 0/0/1 up > > And I add disks as osd by following manual commands: > sudo mkfs -t xfs -f /dev/sdb > sudo mkdir /var/lib/ceph/osd/ceph-1 > sudo mount /dev/sdb /var/lib/ceph/osd/ceph-1/ > sudo ceph-osd -i 1 --mkfs --mkkey > ceph osd create > ceph osd crush add osd.1 1.0 host=storage01 > sudo ceph-osd -i 1 > > for 10 times, and I got: > storage at storage01:~/ceph$ ceph osd tree > > # id weight type name up/down reweight > -2 10 host storage01 > 0 1 osd.0 up 1 > 1 1 osd.1 up 1 > 2 1 osd.2 up 1 > 3 1 osd.3 up 1 > 4 1 osd.4 up 1 > 5 1 osd.5 up 1 > 6 1 osd.6 up 1 > 7 1 osd.7 up 1 > 8 1 osd.8 up 1 > 9 1 osd.9 up 1 > -1 0 root default > > and > > storage at storage01:~/ceph$ ceph -s > health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean > monmap e1: 1 mons at {storage01=192.168.11.1:6789/0}, election > epoch 2, quorum 0 storage01 > osdmap e32: 10 osds: 10 up, 10 in > pgmap v56: 192 pgs: 192 creating; 0 bytes data, 10565 MB used, > 37231 GB / 37242 GB avail > mdsmap e1: 0/0/1 up > > I use the same method to install on storage02, copy > /etc/ceph/ceph.conf from storage01 and use same method to utilize osd > 10~19 > > And I got > storage at storage02:~/ceph$ ceph osd tree > > # id weight type name up/down reweight > -3 10 host storage02 > 10 1 osd.10 up 1 > 11 1 osd.11 up 1 > 12 1 osd.12 up 1 > 13 1 osd.13 up 1 > 14 1 osd.14 up 1 > 15 1 osd.15 up 1 > 16 1 osd.16 up 1 > 17 1 osd.17 up 1 > 18 1 osd.18 up 1 > 19 1 osd.19 up 1 > -2 10 host storage01 > 0 1 osd.0 up 1 > 1 1 osd.1 up 1 > 2 1 osd.2 up 1 > 3 1 osd.3 up 1 > 4 1 osd.4 up 1 > 5 1 osd.5 up 1 > 6 1 osd.6 up 1 > 7 1 osd.7 up 1 > 8 1 osd.8 up 1 > 9 1 osd.9 up 1 > -1 0 root default > > storage at storage02:~/ceph$ ceph -s > health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean > monmap e1: 1 mons at {storage01=192.168.11.1:6789/0}, election > epoch 2, quorum 0 storage01 > osdmap e63: 20 osds: 20 up, 20 in > pgmap v138: 192 pgs: 192 creating; 0 bytes data, 21140 MB used, > 74463 GB / 74484 GB avail > mdsmap e1: 0/0/1 up > > Nothing in progress, it just kept at creating status. How can I debug this? Assuming there isn't some other issue, this is caused by your crush map configuration. With the default rules, CRUSH will start out at the "root" node and try to find OSDs under that, but in your map there's nothing under the "root" node. If you link up the hosts underneath the root I think things should start working. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com
 |