Hi Guido,
On 11/25/2011 05:53 PM, Guido Winkelmann wrote:
Hi,
When trying to mount a Ceph filesystem with "mount.ceph 10.3.1.33:6789:/ /mnt",
the command hangs for several minutes and then fails with the message
mount error 5 = Input/output error
Does "ceph -s" work? If so, what is the output?
Background:
I'm trying to set up a small 3-machine Ceph cluster, to be used for network-
transparent block- and file storage for Qemu server virtualization. This is the
first time I'm doing anything at all with Ceph, so many of the concepts are
still new and a bit confusing to me.
My plan was to set up each of the three machines equally with one mon, one osd
and one mds, and to add more servers, or replace the existing ones with bigger
machines, as need arises.
If should be enough to have 3 OSD's and 1 MON and 1 MDS, for basic
testing that is all you'd need.
So far, I have only started with one of the three servers, to get at least a
basic configuration working and to get some feel for the system and its
components, before adding the other two.
The machines are all running a basic installation of CentOS 6. What I did so
far on the first machine was:
- Build a newer kernel (3.1.1) with built-in support for btrfs and ceph,
reboot with it
- Prepare two larger empty partitions (sda5 and sdb5) for data storage
- Download and unpack ceph-0.38.tar.gz, cd ceph-0.38
- Run ./configure
- Repeat numerous times while installing one dependency after the other,
eventually ending up with ./configure --without-fuse --without-tcmalloc
I'd recommend using tcmalloc, it will save a lot of memory usage.
- make&& make install
- Copy src/init-ceph to /etc/init.d/
- Create /usr/local/etc/ceph/ceph.conf with this content:
[global]
max open files = 131072
log file = /var/log/ceph/$name.log
log_to_syslog = true
pid file = /var/run/ceph/$name.pid
You want to log to syslog and files simultaneously? That will cause
double the I/O on that system.
[mon]
mon data = /mondata/$name
[mon.ceph1]
host = ceph1
mon addr = 10.3.1.33:6789
[mds]
keyring = /cephxdata/keyring.$name
Why did you specify a keyring here? You did not enable the cephx
authentication.
[mds.alpha]
host = ceph1
[osd]
osd data = /data/$name
osd journal = /data/$name/journal
osd journal size = 1000 ; journal size, in megabytes
[osd.0]
host = ceph1
btrfs devs = /dev/sda5 /dev/sdb5
Does that actually work? Never tested with specifying to devices. What
are you trying to do? Create a striped filesystem?
(Slightly adjusted from src/sample.ceph.conf, comments removed for brevity)
- Run mkcephfs -c /usr/local/etc/ceph/ceph.conf --mkbtrfs -a -k \
/usr/local/etc/ceph/keyring.bin
- Run /etc/init.d/ceph start
After these steps, I tried to mount the ceph filesystem (on the same machine)
with "mount.ceph 10.3.1.33:6789:/ /mnt" and got the aforementioned error.
Mounting the Ceph filesystem on the same host as where your OSD is
running is not recommended. Although it should probably work, you could
run into some trouble.
You could take a look at my ceph.conf: http://zooi.widodh.nl/ceph/ceph.conf
That might give you some more clues!
Wido
The only thing that happens in the logs when I try to mount is this one line
in mon log:
2011-11-25 17:26:53.216745 7f0ee75df700 -- 10.3.1.33:6789/0>>
10.3.1.33:0/1719960241 pipe(0x7f0ee0001320 sd=12 pgs=0 cs=0 l=0).accept peer
addr is really 10.3.1.33:0/1719960241 (socket is 10.3.1.33:46908/0)
The osd and mds logs do not show any activity at that point.
Does anybody have any idea what could be going wrong here?
Guido
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
