Hello Gregory: I'm doing some comparison about performance between different combination of environment. Therefore I have to try such old version. Thanks for your kindly help! The solution you provided does work! I think I was relying on ceph-disk too much therefore I didn't noticed this. 2014-08-20 1:44 GMT+08:00 Gregory Farnum <greg at inktank.com>: > On Thu, Aug 14, 2014 at 2:28 AM, NotExist <notexist at gmail.com> wrote: >> Hello everyone: >> >> Since there's no cuttlefish package for 14.04 server on ceph >> repository (only ceph-deploy there), I tried to build cuttlefish from >> source on 14.04. > > ...why? Cuttlefish is old and no longer provided updates. You really > want to be using either Dumpling or Firefly. > >> >> Here's what I did: >> Get source by following http://ceph.com/docs/master/install/clone-source/ >> Enter the sourcecode directory >> git checkout cluttlefish >> git submodule update >> rm -rf src/civetweb/ src/erasure-code/ src/rocksdb/ >> to get the latest cluttlefish repo. >> >> Build source by following http://ceph.com/docs/master/install/build-ceph/ >> beside the package this url mentioned for Ubuntu: >> >> sudo apt-get install autotools-dev autoconf automake cdbs gcc g++ git >> libboost-dev libedit-dev libssl-dev libtool libfcgi libfcgi-dev >> libfuse-dev linux-kernel-headers libcrypto++-dev libcrypto++ >> libexpat1-dev pkg-config >> sudo apt-get install uuid-dev libkeyutils-dev libgoogle-perftools-dev >> libatomic-ops-dev libaio-dev libgdata-common libgdata13 libsnappy-dev >> libleveldb-dev >> >> I also found it will need >> >> sudo apt-get install libboost-filesystem-dev libboost-thread-dev >> libboost-program-options-dev >> >> (And xfsprogs if you need xfs) >> after all packages are installed, I start to complie according to the doc: >> >> ./autogen.sh >> ./configure >> make -j8 >> >> And install following >> http://ceph.com/docs/master/install/install-storage-cluster/#installing-a-build >> >> sudo make install >> >> everything seems fine, but I found ceph_common.sh had been putted to >> '/usr/local/lib/ceph', and some tools are putted into >> /usr/local/usr/local/sbin/ (ceph-disk* and ceph-create-keys). I was >> used to use ceph-disk to prepare the disk on other deployment (on >> other machines with Emperor), but I can't do it now (and maybe the >> path is the reason) so I choose to do do all stuffs manually. >> >> I follow the doc >> http://ceph.com/docs/master/install/manual-deployment/ to deploy the >> cluster many times, but it turns out different this time. >> /etc/ceph isn't there, therefore I sudo mkdir /etc/ceph >> Put a ceph.conf into /etc/ceph >> Generate all required keys in /etc/ceph instead of /tmp/ to keep them >> >> ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n >> mon. --cap mon 'allow *' >> ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring >> --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd >> 'allow *' --cap mds 'allow' >> ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring >> /etc/ceph/ceph.client.admin.keyring >> >> Generate monmap with monmaptool >> >> monmaptool --create --add storage01 192.168.11.1 --fsid >> 9f8fffe3-040d-4641-b35a-ffa90241f723 /etc/ceph/monmap >> >> /var/lib/ceph is not there either >> >> sudo mkdir -p /var/lib/ceph/mon/ceph-storage01 >> sudo ceph-mon --mkfs -i storage01 --monmap /etc/ceph/monmap --keyring >> /etc/ceph/ceph.mon.keyring >> >> log directory are not there, so I create it manually: >> >> sudo mkdir /var/log/ceph >> >> since service doesn't work, I start mon daemon manually: >> >> sudo /usr/local/bin/ceph-mon -i storage01 >> >> and ceph -s looks like these: >> storage at storage01:~/ceph$ ceph -s >> health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds >> monmap e1: 1 mons at {storage01=192.168.11.1:6789/0}, election >> epoch 2, quorum 0 storage01 >> osdmap e1: 0 osds: 0 up, 0 in >> pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail >> mdsmap e1: 0/0/1 up >> >> And I add disks as osd by following manual commands: >> sudo mkfs -t xfs -f /dev/sdb >> sudo mkdir /var/lib/ceph/osd/ceph-1 >> sudo mount /dev/sdb /var/lib/ceph/osd/ceph-1/ >> sudo ceph-osd -i 1 --mkfs --mkkey >> ceph osd create >> ceph osd crush add osd.1 1.0 host=storage01 >> sudo ceph-osd -i 1 >> >> for 10 times, and I got: >> storage at storage01:~/ceph$ ceph osd tree >> >> # id weight type name up/down reweight >> -2 10 host storage01 >> 0 1 osd.0 up 1 >> 1 1 osd.1 up 1 >> 2 1 osd.2 up 1 >> 3 1 osd.3 up 1 >> 4 1 osd.4 up 1 >> 5 1 osd.5 up 1 >> 6 1 osd.6 up 1 >> 7 1 osd.7 up 1 >> 8 1 osd.8 up 1 >> 9 1 osd.9 up 1 >> -1 0 root default >> >> and >> >> storage at storage01:~/ceph$ ceph -s >> health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean >> monmap e1: 1 mons at {storage01=192.168.11.1:6789/0}, election >> epoch 2, quorum 0 storage01 >> osdmap e32: 10 osds: 10 up, 10 in >> pgmap v56: 192 pgs: 192 creating; 0 bytes data, 10565 MB used, >> 37231 GB / 37242 GB avail >> mdsmap e1: 0/0/1 up >> >> I use the same method to install on storage02, copy >> /etc/ceph/ceph.conf from storage01 and use same method to utilize osd >> 10~19 >> >> And I got >> storage at storage02:~/ceph$ ceph osd tree >> >> # id weight type name up/down reweight >> -3 10 host storage02 >> 10 1 osd.10 up 1 >> 11 1 osd.11 up 1 >> 12 1 osd.12 up 1 >> 13 1 osd.13 up 1 >> 14 1 osd.14 up 1 >> 15 1 osd.15 up 1 >> 16 1 osd.16 up 1 >> 17 1 osd.17 up 1 >> 18 1 osd.18 up 1 >> 19 1 osd.19 up 1 >> -2 10 host storage01 >> 0 1 osd.0 up 1 >> 1 1 osd.1 up 1 >> 2 1 osd.2 up 1 >> 3 1 osd.3 up 1 >> 4 1 osd.4 up 1 >> 5 1 osd.5 up 1 >> 6 1 osd.6 up 1 >> 7 1 osd.7 up 1 >> 8 1 osd.8 up 1 >> 9 1 osd.9 up 1 >> -1 0 root default >> >> storage at storage02:~/ceph$ ceph -s >> health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean >> monmap e1: 1 mons at {storage01=192.168.11.1:6789/0}, election >> epoch 2, quorum 0 storage01 >> osdmap e63: 20 osds: 20 up, 20 in >> pgmap v138: 192 pgs: 192 creating; 0 bytes data, 21140 MB used, >> 74463 GB / 74484 GB avail >> mdsmap e1: 0/0/1 up >> >> Nothing in progress, it just kept at creating status. How can I debug this? > > Assuming there isn't some other issue, this is caused by your crush > map configuration. With the default rules, CRUSH will start out at the > "root" node and try to find OSDs under that, but in your map there's > nothing under the "root" node. If you link up the hosts underneath the > root I think things should start working. > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com
 |