Re: Ceph on Raspberry Pi

[Shlomo Dubrowin <shlomo@xxxxxxxxxxxx>]

OK, 

So I'll need to do the installation manually, but the rest of the commands I should run via ceph-deploy?  What version should I be trying to grab for the manual compilation?  Should I be grabbing from git or is there a better place?

  Shlomo

-----------------------------
Shlomo Dubrowin

The Solution to the water crisis in Israel:

# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
  cat /sea/med /sea/red |\
  grep -vi "salt" |\
  tee /sea/dead /lake/kinneret
  LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
done

On Tue, Dec 3, 2013 at 5:25 PM, Alfredo Deza <alfredo.deza@xxxxxxxxxxx> wrote:

On Tue, Dec 3, 2013 at 10:21 AM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote:
> Guys, I don't think we have pre-released packages of anything new that is
> going to work on the pi regardless if you use ceph-deploy.  Look at our
> armhf packages file:
>
> http://ceph.com/debian-emperor/dists/wheezy/main/binary-armhf/Packages
>
> Unless I'm mistaken, you're going to have to compile it yourself.  I think
> Joao was going to try that, not sure if he ever got around to it though.

Oh, good point. I was just assuming that because we support the Distro
the packages would exist.

>
> Mark
>
>
> On 12/03/2013 09:03 AM, Alfredo Deza wrote:
>>
>> On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin <shlomo@xxxxxxxxxxxx>
>> wrote:
>>>
>>> Alfredo,
>>>
>>> I started that way, but I run into an error:
>>>
>>> $ ceph-deploy install baxter
>>> [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy install
>>> baxter
>>> [ceph_deploy.install][DEBUG ] Installing stable version emperor on
>>> cluster
>>> ceph hosts baxter
>>> [ceph_deploy.install][DEBUG ] Detecting platform for host baxter ...
>>> [baxter][DEBUG ] connected to host: baxter
>>> [baxter][DEBUG ] detect platform information from remote host
>>> [baxter][DEBUG ] detect machine type
>>> [ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
>>> [baxter][INFO  ] installing ceph on baxter
>>> [baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
>>> apt-get -q install --assume-yes ca-certificates
>>> [baxter][DEBUG ] Reading package lists...
>>> [baxter][DEBUG ] Building dependency tree...
>>> [baxter][DEBUG ] Reading state information...
>>> [baxter][DEBUG ] ca-certificates is already the newest version.
>>> [baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 85 not
>>> upgraded.
>>> [baxter][INFO  ] Running command: sudo wget -O release.asc
>>> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
>>> [baxter][WARNIN] --2013-12-03 16:32:45--
>>> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
>>> [baxter][WARNIN] Resolving ceph.com (ceph.com)... 208.113.241.137
>>> [baxter][WARNIN] Connecting to ceph.com
>>> (ceph.com)|208.113.241.137|:443...
>>> connected.
>>> [baxter][WARNIN] HTTP request sent, awaiting response... 200 OK
>>> [baxter][WARNIN] Length: unspecified [text/plain]
>>> [baxter][WARNIN] Saving to: `release.asc'
>>> [baxter][WARNIN]
>>> [baxter][WARNIN]      0K .
>>> 1.06M=0.002s
>>> [baxter][WARNIN]
>>> [baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) - `release.asc' saved
>>> [1752]
>>> [baxter][WARNIN]
>>> [baxter][INFO  ] Running command: sudo apt-key add release.asc
>>> [baxter][DEBUG ] OK
>>> [baxter][DEBUG ] add ceph deb repo to sources.list
>>> [baxter][INFO  ] Running command: sudo apt-get -q update
>>> [baxter][DEBUG ] Get:1 http://mirrordirector.raspbian.org wheezy
>>> Release.gpg
>>> [490 B]
>>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release.gpg
>>> [baxter][DEBUG ] Get:2 http://archive.raspberrypi.org wheezy Release.gpg
>>> [490 B]
>>> [baxter][DEBUG ] Get:3 http://mirrordirector.raspbian.org wheezy Release
>>> [14.4 kB]
>>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release
>>> [baxter][DEBUG ] Get:4 http://ceph.com wheezy Release.gpg [836 B]
>>> [baxter][DEBUG ] Get:5 http://archive.raspberrypi.org wheezy Release
>>> [7224
>>> B]
>>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy/rpi armhf
>>> Packages
>>> [baxter][DEBUG ] Get:6 http://mirrordirector.raspbian.org wheezy/main
>>> armhf
>>> Packages [7414 kB]
>>> [baxter][DEBUG ] Get:7 http://archive.raspberrypi.org wheezy/main armhf
>>> Packages [12.1 kB]
>>> [baxter][DEBUG ] Get:8 http://ceph.com wheezy Release [5984 B]
>>> [baxter][DEBUG ] Ign http://raspberrypi.collabora.com wheezy/rpi
>>> Translation-en
>>> [baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf Packages [1010
>>> B]
>>> [baxter][DEBUG ] Ign http://archive.raspberrypi.org wheezy/main
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://ceph.com wheezy/main Translation-en
>>> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/contrib
>>> armhf
>>> Packages
>>> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/non-free
>>> armhf Packages
>>> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/rpi armhf
>>> Packages
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/contrib
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/main
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/non-free
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/rpi
>>> Translation-en
>>> [baxter][DEBUG ] Fetched 7456 kB in 47s (157 kB/s)
>>> [baxter][DEBUG ] Reading package lists...
>>> [baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
>>> DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew
>>> --no-install-recommends --assume-yes install -- ceph ceph-mds ceph-common
>>> ceph-fs-common gdisk
>>> [baxter][WARNIN] E: Unable to locate package ceph-mds
>>> [baxter][WARNIN] E: Unable to locate package ceph-fs-common
>>> [baxter][DEBUG ] Reading package lists...
>>> [baxter][DEBUG ] Building dependency tree...
>>> [baxter][DEBUG ] Reading state information...
>>> [baxter][ERROR ] RuntimeError: command returned non-zero exit status: 100
>>> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env
>>> DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o
>>> Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes
>>> install
>>> -- ceph ceph-mds ceph-common ceph-fs-common gdisk
>>
>>
>> That output looks unexpected. I wonder if it is just a network hiccup
>> to fail to get to those two packages.
>>>
>>>
>>>
>>> When running manually:
>>>
>>> $ env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q
>>> -o
>>> Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes
>>> install
>>> -- ceph ceph-mds ceph-common ceph-fs-common gdisk
>>> E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission
>>> denied)
>>> E: Unable to lock the administration directory (/var/lib/dpkg/), are you
>>> root?
>>>
>>>
>>> Maybe this is supposed to be a sudo command?
>>
>>
>> Yes, always with sudo unless you are root, which is the reason why
>> ceph-deploy will not use that in the output.
>>
>>>
>>> $ sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical
>>> apt-get
>>> -q -o Dpkg::Options::=--force-confnew --no-install-recommends
>>> --assume-yes
>>> install -- ceph ceph-mds ceph-common ceph-fs-common gdisk
>>> Reading package lists...
>>> Building dependency tree...
>>> Reading state information...
>>> E: Unable to locate package ceph-mds
>>> E: Unable to locate package ceph-fs-common
>>>
>>>
>>> This was the stage where I tried to install ceph with just apt-get
>>> install
>>> manually.
>>
>>
>> If you run a `sudo apt-get update` does that come back correctly or
>> does it complain?
>>
>>>
>>>    Shlomo
>>>
>>>
>>> -----------------------------
>>> Shlomo Dubrowin
>>>
>>> The Solution to the water crisis in Israel:
>>>
>>> # According to WikiPedia, the Kinneret can hold
>>> # 4 km^3, so FULL here is in cubit meters
>>> FULL="4000"
>>> while [ "$LEVEL" -lt "$FULL" ]; do
>>>    cat /sea/med /sea/red |\
>>>    grep -vi "salt" |\
>>>    tee /sea/dead /lake/kinneret
>>>    LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
>>> done
>>>
>>>
>>> On Tue, Dec 3, 2013 at 4:29 PM, Alfredo Deza <alfredo.deza@xxxxxxxxxxx>
>>> wrote:
>>>>
>>>>
>>>> On Tue, Dec 3, 2013 at 9:21 AM, Shlomo Dubrowin <shlomo@xxxxxxxxxxxx>
>>>> wrote:
>>>>>
>>>>> Alfredo,
>>>>>
>>>>> Thank you for your response.  I simply did apt-get install ceph on the
>>>>> nodes.
>>>>>
>>>>> My /etc/apt/sources.list.d/ceph.list contains:
>>>>>
>>>>> deb http://ceph.com/debian-emperor/ wheezy main
>>>>>
>>>>
>>>> Was that added manually? ceph-deploy can take care of handling the
>>>> sources list for you.
>>>>
>>>> It is possible that you have something else in that machine that is
>>>> saying that 0.43 is the version you should
>>>> be getting.
>>>>
>>>> Have you tried to install with ceph-deploy?:
>>>>
>>>>      ceph-deploy install baxter
>>>>
>>>>
>>>>
>>>>>
>>>>> and the versions I received are what I got.
>>>>>
>>>>>    Shlomo
>>>>>
>>>>> -----------------------------
>>>>> Shlomo Dubrowin
>>>>>
>>>>> The Solution to the water crisis in Israel:
>>>>>
>>>>> # According to WikiPedia, the Kinneret can hold
>>>>> # 4 km^3, so FULL here is in cubit meters
>>>>> FULL="4000"
>>>>> while [ "$LEVEL" -lt "$FULL" ]; do
>>>>>    cat /sea/med /sea/red |\
>>>>>    grep -vi "salt" |\
>>>>>    tee /sea/dead /lake/kinneret
>>>>>    LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print
>>>>> $1}'`
>>>>> done
>>>>>
>>>>>
>>>>> On Tue, Dec 3, 2013 at 4:15 PM, Alfredo Deza <alfredo.deza@xxxxxxxxxxx>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On Tue, Dec 3, 2013 at 8:55 AM, Shlomo Dubrowin <shlomo@xxxxxxxxxxxx>
>>>>>> wrote:
>>>>>>>
>>>>>>> I'm trying to deploy Ceph on a group of Raspberry Pis using the
>>>>>>> procedure
>>>>>>> documented in: http://ceph.com/docs/master/start/quick-ceph-deploy/
>>>>>>>
>>>>>>> There used to be a site:
>>>>>>> http://ceph.com/docs/master/start/quick-start/
>>>>>>> but
>>>>>>> that page is no longer valid.
>>>>>>>
>>>>>>> The first thing I noticed is that the command lsb_release -sc
>>>>>>> specified
>>>>>>> in
>>>>>>> the Ceph Deploy setup provides n/a even though the answer should be
>>>>>>> wheezy.
>>>>>>> I manually changed the /etc/apt/sources.list.d/ceph.list to specify
>>>>>>> wheezy.
>>>>>>>
>>>>>>> I've installed ceph-deploy and setup the keys to communicate between
>>>>>>> the
>>>>>>> deploy Raspberry Pi and all the Raspberry Pis including the system
>>>>>>> reunning
>>>>>>> ceph-deploy.
>>>>>>>
>>>>>>> When I tried to run ceph-deploy install <node1> the installation
>>>>>>> failed.
>>>>>>> I went to each node and installed ceph manually and I see the
>>>>>>> versions
>>>>>>> are:
>>>>>>>
>>>>>>> $ ceph-deploy --version
>>>>>>> 1.3.3
>>>>>>>
>>>>>>> $ ceph --version
>>>>>>> ceph version 0.43 (commit:9fa8781c0147d66fcef7c2dd0e09cd3c69747d37)
>>>>>>>
>>>>>>>
>>>>>>> All the nodes have the same ceph version.
>>>>>>
>>>>>>
>>>>>> That looks like a *very* old ceph version. Is there any reason you are
>>>>>> using 0.43 as opposed to the latest one?
>>>>>>
>>>>>> How did you installed Ceph on those nodes?
>>>>>>>
>>>>>>>
>>>>>>> When I try to run a command, I get errors:
>>>>>>>
>>>>>>> $ ceph-deploy mon create baxter
>>>>>>> [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
>>>>>>> create
>>>>>>> baxter
>>>>>>> [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts baxter
>>>>>>> [ceph_deploy.mon][DEBUG ] detecting platform for host baxter ...
>>>>>>> [baxter][DEBUG ] connected to host: baxter
>>>>>>> [baxter][DEBUG ] detect platform information from remote host
>>>>>>> [baxter][DEBUG ] detect machine type
>>>>>>> [ceph_deploy.mon][INFO  ] distro info: debian 7.0 wheezy
>>>>>>> [baxter][DEBUG ] determining if provided host has same hostname in
>>>>>>> remote
>>>>>>> [baxter][DEBUG ] get remote short hostname
>>>>>>> [baxter][DEBUG ] deploying mon to baxter
>>>>>>> [baxter][DEBUG ] get remote short hostname
>>>>>>> [baxter][DEBUG ] remote hostname: baxter
>>>>>>> [baxter][DEBUG ] write cluster configuration to
>>>>>>> /etc/ceph/{cluster}.conf
>>>>>>> [baxter][DEBUG ] create the mon path if it does not exist
>>>>>>> [baxter][DEBUG ] checking for done path:
>>>>>>> /var/lib/ceph/mon/ceph-baxter/done
>>>>>>> [baxter][DEBUG ] done path does not exist:
>>>>>>> /var/lib/ceph/mon/ceph-baxter/done
>>>>>>> [baxter][INFO  ] creating keyring file:
>>>>>>> /var/lib/ceph/tmp/ceph-baxter.mon.keyring
>>>>>>> [baxter][DEBUG ] create the monitor keyring file
>>>>>>> [baxter][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs
>>>>>>> -i
>>>>>>> baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
>>>>>>> [baxter][WARNIN] too many arguments: [--cluster,ceph]
>>>>>>> [baxter][WARNIN] usage: ceph-mon -i monid [--mon-data=""> >>>>>>> [flags]
>>>>>>> [baxter][WARNIN]   --debug_mon n
>>>>>>> [baxter][WARNIN]         debug monitor level (e.g. 10)
>>>>>>> [baxter][WARNIN]   --mkfs
>>>>>>> [baxter][WARNIN]         build fresh monitor fs
>>>>>>> [baxter][DEBUG ] --conf/-c        Read configuration from the given
>>>>>>> configuration file
>>>>>>> [baxter][DEBUG ] -d               Run in foreground, log to stderr.
>>>>>>> [baxter][DEBUG ] -f               Run in foreground, log to usual
>>>>>>> location.
>>>>>>> [baxter][DEBUG ] --id/-i          set ID portion of my name
>>>>>>> [baxter][DEBUG ] --name/-n        set name (TYPE.ID)
>>>>>>> [baxter][DEBUG ] --version        show version and quit
>>>>>>> [baxter][DEBUG ]
>>>>>>> [baxter][DEBUG ]    --debug_ms N
>>>>>>> [baxter][DEBUG ]         set message debug level (e.g. 1)
>>>>>>> [baxter][ERROR ] RuntimeError: command returned non-zero exit status:
>>>>>>> 1
>>>>>>> [ceph_deploy.mon][ERROR ] Failed to execute command: ceph-mon
>>>>>>> --cluster
>>>>>>> ceph
>>>>>>> --mkfs -i baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
>>>>>>> [ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
>>>>>>>
>>>>>>>
>>>>>>> If I try to run the same command as the user ceph on the local
>>>>>>> machine,
>>>>>>> I
>>>>>>> get the same error:
>>>>>>>
>>>>>>> $ ceph-mon --cluster ceph --mkfs -i baxter --keyring
>>>>>>> /var/lib/ceph/tmp/ceph-baxter.mon.keyring
>>>>>>> failed to open log file '/var/log/ceph/mon.baxter.log': (13)
>>>>>>> Permission
>>>>>>> denied
>>>>>>> too many arguments: [--cluster,ceph]
>>>>>>> usage: ceph-mon -i monid [--mon-data="" [flags]
>>>>>>>    --debug_mon n
>>>>>>>          debug monitor level (e.g. 10)
>>>>>>>    --mkfs
>>>>>>>          build fresh monitor fs
>>>>>>> --conf/-c        Read configuration from the given configuration file
>>>>>>> -d               Run in foreground, log to stderr.
>>>>>>> -f               Run in foreground, log to usual location.
>>>>>>> --id/-i          set ID portion of my name
>>>>>>> --name/-n        set name (TYPE.ID)
>>>>>>> --version        show version and quit
>>>>>>>
>>>>>>>     --debug_ms N
>>>>>>>          set message debug level (e.g. 1)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This makes me think that the ceph-deploy installed doesn't work with
>>>>>>> the
>>>>>>> ceph installed on each node.
>>>>>>>
>>>>>>> So I guess I have 2 questions:
>>>>>>>
>>>>>>> How can I determine which ceph-deploy goes with which ceph and how
>>>>>>> can I
>>>>>>> correct this apparent mismatch?
>>>>>>
>>>>>>
>>>>>> I don't think there was a ceph-deploy per-se that would've worked for
>>>>>> that Ceph version.
>>>>>>
>>>>>> I believe that ceph-deploy is very backwards compatible for a few Ceph
>>>>>> versions. At the very least we should
>>>>>> fully support the latest 3 major versions and we special case a few
>>>>>> commands depending on the version installed.
>>>>>>
>>>>>> But like I mentioned, the Ceph version you have installed seems like a
>>>>>> very very old one.
>>>>>>
>>>>>>> Should I be looking at just using local ceph commands to configure
>>>>>>> this
>>>>>>> by
>>>>>>> hand? Are there instructions on how to do this by hand?
>>>>>>
>>>>>>
>>>>>> If you must have that version I don't think there is any option here
>>>>>> but to do it by hand and avoid ceph-deploy.
>>>>>>
>>>>>> For docs, I think that might be even more difficult to do as you would
>>>>>> need to go to the tag for that version and look at the docs
>>>>>> for that tag in the repository.
>>>>>>
>>>>>> Then again, what is the reason for using that version?
>>>>>>
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>>    Shlomo
>>>>>>>
>>>>>>> -----------------------------
>>>>>>> Shlomo Dubrowin
>>>>>>>
>>>>>>> The Solution to the water crisis in Israel:
>>>>>>>
>>>>>>> # According to WikiPedia, the Kinneret can hold
>>>>>>> # 4 km^3, so FULL here is in cubit meters
>>>>>>> FULL="4000"
>>>>>>> while [ "$LEVEL" -lt "$FULL" ]; do
>>>>>>>    cat /sea/med /sea/red |\
>>>>>>>    grep -vi "salt" |\
>>>>>>>    tee /sea/dead /lake/kinneret
>>>>>>>    LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print
>>>>>>> $1}'`
>>>>>>> done
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com