On 12/03/2013 03:38 PM, Shlomo Dubrowin wrote:
Joao,
Is there a reason you aren't putting OSDs on the Pis? Do you expect that
the OSD won't run on the Pi?
The question really isn't about why I'm not putting the OSDs on the pi's
but why I'd prefer to put them on the Cubietruck: unlike the pi,
cubietruck has a SATA connection.
Besides, yeah, the Pi doesn't have much RAM anyway, and although I'm
doing it just for the kicks of it and don't aim at a performing cluster,
I don't see the 512MB of RAM to be enough for a mon, an OSD and the
overall OS.
-Joao
Shlomo
-----------------------------
Shlomo Dubrowin
The Solution to the water crisis in Israel:
# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
cat /sea/med /sea/red |\
grep -vi "salt" |\
tee /sea/dead /lake/kinneret
LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
done
On Tue, Dec 3, 2013 at 5:33 PM, Joao Eduardo Luis <joao.luis@xxxxxxxxxxx
<mailto:joao.luis@xxxxxxxxxxx>> wrote:
On 12/03/2013 03:21 PM, Mark Nelson wrote:
Guys, I don't think we have pre-released packages of anything
new that
is going to work on the pi regardless if you use ceph-deploy.
Look at
our armhf packages file:
http://ceph.com/debian-__emperor/dists/wheezy/main/__binary-armhf/Packages
<http://ceph.com/debian-emperor/dists/wheezy/main/binary-armhf/Packages>
Unless I'm mistaken, you're going to have to compile it yourself. I
think Joao was going to try that, not sure if he ever got around
to it
though.
I got as far as hitting some missing leveldb dependency.
Then decided to distcc and cross-compile Ceph using my desktop, but
messed the toolchain somehow and have been attending to figuring out
that every now and then but to no avail yet.
My best guess atm is that regardless of being able to compile it on
the pi, we're going to hit some missing dependencies anyway
rendering the whole task mighty annoying. I'm hoping to be able to
cross-compile a static version of the binaries and have a couple of
monitors on the pi and an OSD on a cubietruck by Christmas :)
-Joao
Mark
On 12/03/2013 09:03 AM, Alfredo Deza wrote:
On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin
<shlomo@xxxxxxxxxxxx <mailto:shlomo@xxxxxxxxxxxx>>
wrote:
Alfredo,
I started that way, but I run into an error:
$ ceph-deploy install baxter
[ceph_deploy.cli][INFO ] Invoked (1.3.3):
/usr/bin/ceph-deploy install
baxter
[ceph_deploy.install][DEBUG ] Installing stable version
emperor on
cluster
ceph hosts baxter
[ceph_deploy.install][DEBUG ] Detecting platform for
host baxter ...
[baxter][DEBUG ] connected to host: baxter
[baxter][DEBUG ] detect platform information from remote
host
[baxter][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: debian 7.0 wheezy
[baxter][INFO ] installing ceph on baxter
[baxter][INFO ] Running command: sudo env
DEBIAN_FRONTEND=noninteractive
apt-get -q install --assume-yes ca-certificates
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][DEBUG ] ca-certificates is already the newest
version.
[baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to
remove and 85 not
upgraded.
[baxter][INFO ] Running command: sudo wget -O release.asc
https://ceph.com/git/?p=ceph.__git;a=blob_plain;f=keys/__release.asc
<https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc>
[baxter][WARNIN] --2013-12-03 16:32:45--
https://ceph.com/git/?p=ceph.__git;a=blob_plain;f=keys/__release.asc
<https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc>
[baxter][WARNIN] Resolving ceph.com <http://ceph.com>
(ceph.com <http://ceph.com>)... 208.113.241.137
[baxter][WARNIN] Connecting to ceph.com <http://ceph.com>
(ceph.com <http://ceph.com>)|208.113.241.137|:__443...
connected.
[baxter][WARNIN] HTTP request sent, awaiting response...
200 OK
[baxter][WARNIN] Length: unspecified [text/plain]
[baxter][WARNIN] Saving to: `release.asc'
[baxter][WARNIN]
[baxter][WARNIN] 0K .
1.06M=0.002s
[baxter][WARNIN]
[baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) -
`release.asc' saved
[1752]
[baxter][WARNIN]
[baxter][INFO ] Running command: sudo apt-key add
release.asc
[baxter][DEBUG ] OK
[baxter][DEBUG ] add ceph deb repo to sources.list
[baxter][INFO ] Running command: sudo apt-get -q update
[baxter][DEBUG ] Get:1
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy
Release.gpg
[490 B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.__com
<http://raspberrypi.collabora.com> wheezy Release.gpg
[baxter][DEBUG ] Get:2 http://archive.raspberrypi.org
wheezy Release.gpg
[490 B]
[baxter][DEBUG ] Get:3
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy Release
[14.4 kB]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.__com
<http://raspberrypi.collabora.com> wheezy Release
[baxter][DEBUG ] Get:4 http://ceph.com wheezy
Release.gpg [836 B]
[baxter][DEBUG ] Get:5 http://archive.raspberrypi.org
wheezy Release
[7224
B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.__com
<http://raspberrypi.collabora.com> wheezy/rpi armhf
Packages
[baxter][DEBUG ] Get:6
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy/main
armhf
Packages [7414 kB]
[baxter][DEBUG ] Get:7 http://archive.raspberrypi.org
wheezy/main armhf
Packages [12.1 kB]
[baxter][DEBUG ] Get:8 http://ceph.com wheezy Release
[5984 B]
[baxter][DEBUG ] Ign http://raspberrypi.collabora.__com
<http://raspberrypi.collabora.com> wheezy/rpi
Translation-en
[baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf
Packages
[1010 B]
[baxter][DEBUG ] Ign http://archive.raspberrypi.org
wheezy/main
Translation-en
[baxter][DEBUG ] Ign http://ceph.com wheezy/main
Translation-en
[baxter][DEBUG ] Hit
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org>
wheezy/contrib armhf
Packages
[baxter][DEBUG ] Hit
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy/non-free
armhf Packages
[baxter][DEBUG ] Hit
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy/rpi armhf
Packages
[baxter][DEBUG ] Ign
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy/contrib
Translation-en
[baxter][DEBUG ] Ign
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy/main
Translation-en
[baxter][DEBUG ] Ign
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy/non-free
Translation-en
[baxter][DEBUG ] Ign
http://mirrordirector.__raspbian.org
<http://mirrordirector.raspbian.org> wheezy/rpi
Translation-en
[baxter][DEBUG ] Fetched 7456 kB in 47s (157 kB/s)
[baxter][DEBUG ] Reading package lists...
[baxter][INFO ] Running command: sudo env
DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical apt-get -q -o
Dpkg::Options::=--force-__confnew
--no-install-recommends --assume-yes install -- ceph
ceph-mds
ceph-common
ceph-fs-common gdisk
[baxter][WARNIN] E: Unable to locate package ceph-mds
[baxter][WARNIN] E: Unable to locate package ceph-fs-common
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][ERROR ] RuntimeError: command returned non-zero
exit status:
100
[ceph_deploy][ERROR ] RuntimeError: Failed to execute
command: env
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical
apt-get -q -o
Dpkg::Options::=--force-__confnew
--no-install-recommends --assume-yes
install
-- ceph ceph-mds ceph-common ceph-fs-common gdisk
That output looks unexpected. I wonder if it is just a
network hiccup
to fail to get to those two packages.
When running manually:
$ env DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical apt-get
-q -o
Dpkg::Options::=--force-__confnew
--no-install-recommends --assume-yes
install
-- ceph ceph-mds ceph-common ceph-fs-common gdisk
E: Could not open lock file /var/lib/dpkg/lock - open
(13: Permission
denied)
E: Unable to lock the administration directory
(/var/lib/dpkg/), are you
root?
Maybe this is supposed to be a sudo command?
Yes, always with sudo unless you are root, which is the
reason why
ceph-deploy will not use that in the output.
$ sudo env DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical
apt-get
-q -o Dpkg::Options::=--force-__confnew
--no-install-recommends
--assume-yes
install -- ceph ceph-mds ceph-common ceph-fs-common gdisk
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package ceph-mds
E: Unable to locate package ceph-fs-common
This was the stage where I tried to install ceph with
just apt-get
install
manually.
If you run a `sudo apt-get update` does that come back
correctly or
does it complain?
Shlomo
-----------------------------
Shlomo Dubrowin
The Solution to the water crisis in Israel:
# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
cat /sea/med /sea/red |\
grep -vi "salt" |\
tee /sea/dead /lake/kinneret
LEVEL=`du -c /sea/dead /lake/kinneret | grep total |
awk '{print
$1}'`
done
On Tue, Dec 3, 2013 at 4:29 PM, Alfredo Deza
<alfredo.deza@xxxxxxxxxxx <mailto:alfredo.deza@xxxxxxxxxxx>>
wrote:
On Tue, Dec 3, 2013 at 9:21 AM, Shlomo Dubrowin
<shlomo@xxxxxxxxxxxx <mailto:shlomo@xxxxxxxxxxxx>>
wrote:
Alfredo,
Thank you for your response. I simply did
apt-get install ceph on the
nodes.
My /etc/apt/sources.list.d/ceph.__list contains:
deb http://ceph.com/debian-__emperor/
<http://ceph.com/debian-emperor/> wheezy main
Was that added manually? ceph-deploy can take care
of handling the
sources list for you.
It is possible that you have something else in that
machine that is
saying that 0.43 is the version you should
be getting.
Have you tried to install with ceph-deploy?:
ceph-deploy install baxter
and the versions I received are what I got.
Shlomo
-----------------------------
Shlomo Dubrowin
The Solution to the water crisis in Israel:
# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
cat /sea/med /sea/red |\
grep -vi "salt" |\
tee /sea/dead /lake/kinneret
LEVEL=`du -c /sea/dead /lake/kinneret | grep
total | awk '{print
$1}'`
done
On Tue, Dec 3, 2013 at 4:15 PM, Alfredo Deza
<alfredo.deza@xxxxxxxxxxx
<mailto:alfredo.deza@xxxxxxxxxxx>>
wrote:
On Tue, Dec 3, 2013 at 8:55 AM, Shlomo
Dubrowin <shlomo@xxxxxxxxxxxx
<mailto:shlomo@xxxxxxxxxxxx>>
wrote:
I'm trying to deploy Ceph on a group of
Raspberry Pis using the
procedure
documented in:
http://ceph.com/docs/master/__start/quick-ceph-deploy/
<http://ceph.com/docs/master/start/quick-ceph-deploy/>
There used to be a site:
http://ceph.com/docs/master/__start/quick-start/
<http://ceph.com/docs/master/start/quick-start/>
but
that page is no longer valid.
The first thing I noticed is that the
command lsb_release -sc
specified
in
the Ceph Deploy setup provides n/a even
though the answer should be
wheezy.
I manually changed the
/etc/apt/sources.list.d/ceph.__list to
specify
wheezy.
I've installed ceph-deploy and setup the
keys to communicate between
the
deploy Raspberry Pi and all the
Raspberry Pis including the system
reunning
ceph-deploy.
When I tried to run ceph-deploy install
<node1> the installation
failed.
I went to each node and installed ceph
manually and I see the
versions
are:
$ ceph-deploy --version
1.3.3
$ ceph --version
ceph version 0.43
(commit:__9fa8781c0147d66fcef7c2dd0e09cd__3c69747d37)
All the nodes have the same ceph version.
That looks like a *very* old ceph version.
Is there any reason you
are
using 0.43 as opposed to the latest one?
How did you installed Ceph on those nodes?
When I try to run a command, I get errors:
$ ceph-deploy mon create baxter
[ceph_deploy.cli][INFO ] Invoked
(1.3.3): /usr/bin/ceph-deploy mon
create
baxter
[ceph_deploy.mon][DEBUG ] Deploying mon,
cluster ceph hosts baxter
[ceph_deploy.mon][DEBUG ] detecting
platform for host baxter ...
[baxter][DEBUG ] connected to host: baxter
[baxter][DEBUG ] detect platform
information from remote host
[baxter][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info:
debian 7.0 wheezy
[baxter][DEBUG ] determining if provided
host has same hostname in
remote
[baxter][DEBUG ] get remote short hostname
[baxter][DEBUG ] deploying mon to baxter
[baxter][DEBUG ] get remote short hostname
[baxter][DEBUG ] remote hostname: baxter
[baxter][DEBUG ] write cluster
configuration to
/etc/ceph/{cluster}.conf
[baxter][DEBUG ] create the mon path if
it does not exist
[baxter][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-baxter/__done
[baxter][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-baxter/__done
[baxter][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-baxter.__mon.keyring
[baxter][DEBUG ] create the monitor
keyring file
[baxter][INFO ] Running command: sudo
ceph-mon --cluster ceph
--mkfs
-i
baxter --keyring
/var/lib/ceph/tmp/ceph-baxter.__mon.keyring
[baxter][WARNIN] too many arguments:
[--cluster,ceph]
[baxter][WARNIN] usage: ceph-mon -i
monid [--mon-data=pathtodata]
[flags]
[baxter][WARNIN] --debug_mon n
[baxter][WARNIN] debug monitor
level (e.g. 10)
[baxter][WARNIN] --mkfs
[baxter][WARNIN] build fresh
monitor fs
[baxter][DEBUG ] --conf/-c Read
configuration from the given
configuration file
[baxter][DEBUG ] -d Run in
foreground, log to stderr.
[baxter][DEBUG ] -f Run in
foreground, log to usual
location.
[baxter][DEBUG ] --id/-i set ID
portion of my name
[baxter][DEBUG ] --name/-n set
name (TYPE.ID <http://TYPE.ID>)
[baxter][DEBUG ] --version show
version and quit
[baxter][DEBUG ]
[baxter][DEBUG ] --debug_ms N
[baxter][DEBUG ] set message
debug level (e.g. 1)
[baxter][ERROR ] RuntimeError: command
returned non-zero exit
status:
1
[ceph_deploy.mon][ERROR ] Failed to
execute command: ceph-mon
--cluster
ceph
--mkfs -i baxter --keyring
/var/lib/ceph/tmp/ceph-baxter.__mon.keyring
[ceph_deploy][ERROR ] GenericError:
Failed to create 1 monitors
If I try to run the same command as the
user ceph on the local
machine,
I
get the same error:
$ ceph-mon --cluster ceph --mkfs -i
baxter --keyring
/var/lib/ceph/tmp/ceph-baxter.__mon.keyring
failed to open log file
'/var/log/ceph/mon.baxter.log'__: (13)
Permission
denied
too many arguments: [--cluster,ceph]
usage: ceph-mon -i monid
[--mon-data=pathtodata] [flags]
--debug_mon n
debug monitor level (e.g. 10)
--mkfs
build fresh monitor fs
--conf/-c Read configuration from
the given configuration
file
-d Run in foreground, log
to stderr.
-f Run in foreground, log
to usual location.
--id/-i set ID portion of my name
--name/-n set name (TYPE.ID
<http://TYPE.ID>)
--version show version and quit
--debug_ms N
set message debug level (e.g. 1)
This makes me think that the ceph-deploy
installed doesn't work with
the
ceph installed on each node.
So I guess I have 2 questions:
How can I determine which ceph-deploy
goes with which ceph and how
can I
correct this apparent mismatch?
I don't think there was a ceph-deploy per-se
that would've worked for
that Ceph version.
I believe that ceph-deploy is very backwards
compatible for a few
Ceph
versions. At the very least we should
fully support the latest 3 major versions
and we special case a few
commands depending on the version installed.
But like I mentioned, the Ceph version you
have installed seems
like a
very very old one.
Should I be looking at just using local
ceph commands to configure
this
by
hand? Are there instructions on how to
do this by hand?
If you must have that version I don't think
there is any option here
but to do it by hand and avoid ceph-deploy.
For docs, I think that might be even more
difficult to do as you
would
need to go to the tag for that version and
look at the docs
for that tag in the repository.
Then again, what is the reason for using
that version?
Thank you.
Shlomo
-----------------------------
Shlomo Dubrowin
The Solution to the water crisis in Israel:
# According to WikiPedia, the Kinneret
can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
cat /sea/med /sea/red |\
grep -vi "salt" |\
tee /sea/dead /lake/kinneret
LEVEL=`du -c /sea/dead
/lake/kinneret | grep total | awk '{print
$1}'`
done
_________________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
<mailto:ceph-users@xxxxxxxxxxxxxx>
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
_________________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
_________________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
--
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
_________________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
--
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
