On Tue, Nov 12, 2013 at 3:43 PM, Eric Eastman <eric0e@xxxxxxx> wrote: > I built Ceph version 0.72 with --with-libzfs on Ubuntu 1304 after installing > ZFS > from th ppa:zfs-native/stable repository. The ZFS version is v0.6.2-1 > > I do have a few questions and comments on Ceph using ZFS backed OSDs > > As ceph-deploy does not show support for ZFS, I used the instructions at: > http://ceph.com/docs/master/rados/operations/add-or-rm-osds/ > and hand created a new OSD on an existing Ceph system. I guest that I needed > to build a zpool out of a disk, and then create a ZFS file system that > mounted to /var/lib/ceph/osd/ceph-X, where X was the number given when I > ran the ceph osd create command. As I am testing on a VM, I created 2 new > disks, one 2GB (/dev/sde) for journal and one 32GB (/dev/sdd) for data. To > setup the system for ZFS based OSDs, I first added to all my ceph.conf > files: > > filestore zfs_snap = 1 > journal_aio = 0 > journal_dio = 0 > > I then created the OSD with the commands: > > # ceph osd create > 4 > # parted -s /dev/sdd mklabel gpt mkpart -- -- 1 \-1 > # parted -s /dev/sde mklabel gpt mkpart -- -- 1 \-1 > # zpool create sdd /dev/sdd Since you are using the entire disk for your pool, you don't need a GPT label. You can eliminate the parted commands. > # mkdir /var/lib/ceph/osd/ceph-4 > # zfs create -o mountpoint=/var/lib/ceph/osd/ceph-4 sdd/ceph-4 > # ceph-osd -i 4 --mkfs --mkkey --osd-journal=/dev/sde1 --mkjournal > # ceph auth add osd.4 osd 'allow *' mon 'allow rwx' -i > /var/lib/ceph/osd/ceph-4/keyring > > I then decompiled the crush map, added osd.4, and recompiled the map, and > set Ceph to use the new crush map. > > When I started the osd.4 with: > > # start ceph-osd id=4 > > It failed to start, as the ceph osd log file indicated the journal was > missing: > mount failed to open journal /var/lib/ceph/osd/ceph-4/journal: (2) No > such file or directory > > So I manually created a link named journal to /dev/sde1 and created the > journal_uuid file. Should ceph-osd have done this step? Is there anything > else I may of missed? > > With limited testing, the ZFS backed OSD seems to function correctly. > > I was wondering if there are any ZFS file system options that should be set > for better performance or data safety. > > It would be nice if ceph-deploy would handle ZFS. > > Lastly, I want to thank Yan, Zheng and all the rest who worked on this > project. > > Eric > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |