I built Ceph version 0.72 with --with-libzfs on Ubuntu 1304 after
installing ZFS
from th ppa:zfs-native/stable repository. The ZFS version is v0.6.2-1
I do have a few questions and comments on Ceph using ZFS backed OSDs
As ceph-deploy does not show support for ZFS, I used the instructions
at:
http://ceph.com/docs/master/rados/operations/add-or-rm-osds/
and hand created a new OSD on an existing Ceph system. I guest that I
needed to build a zpool out of a disk, and then create a ZFS file
system that mounted to /var/lib/ceph/osd/ceph-X, where X was the
number given when I ran the ceph osd create command. As I am testing
on a VM, I created 2 new disks, one 2GB (/dev/sde) for journal and one
32GB (/dev/sdd) for data. To setup the system for ZFS based OSDs, I
first added to all my ceph.conf files:
filestore zfs_snap = 1
journal_aio = 0
journal_dio = 0
I then created the OSD with the commands:
# ceph osd create
4
# parted -s /dev/sdd mklabel gpt mkpart -- -- 1 \-1
# parted -s /dev/sde mklabel gpt mkpart -- -- 1 \-1
# zpool create sdd /dev/sdd
# mkdir /var/lib/ceph/osd/ceph-4
# zfs create -o mountpoint=/var/lib/ceph/osd/ceph-4 sdd/ceph-4
# ceph-osd -i 4 --mkfs --mkkey --osd-journal=/dev/sde1 --mkjournal
# ceph auth add osd.4 osd 'allow *' mon 'allow rwx' -i
/var/lib/ceph/osd/ceph-4/keyring
I then decompiled the crush map, added osd.4, and recompiled the map,
and set Ceph to use the new crush map.
When I started the osd.4 with:
# start ceph-osd id=4
It failed to start, as the ceph osd log file indicated the journal was
missing:
mount failed to open journal /var/lib/ceph/osd/ceph-4/journal: (2)
No such file or directory
So I manually created a link named journal to /dev/sde1 and created the
journal_uuid file. Should ceph-osd have done this step? Is there
anything else I may of missed?
With limited testing, the ZFS backed OSD seems to function correctly.
I was wondering if there are any ZFS file system options that should be
set for better performance or data safety.
It would be nice if ceph-deploy would handle ZFS.
Lastly, I want to thank Yan, Zheng and all the rest who worked on this
project.
Eric
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
