Off and on for the past 8 months I've been working on setting up a rather sizable Gluster configuration on top of ZFS. I wanted to report back my experiences.
First, I'm running Ubuntu 12.04 with zfsonlinux.org packages. Gluster is installed from the semiosis PPA.
My test config is as follows:
Two brick Gluster in mirroring (replica 2)
Each brick has 8 TB of hard drives configured with ZFS using RAID-Z.
Each brick has a 256 GB SSD drive to use as cache for ZFS.
A third server for geo-replication slave also with RAID-Z storage.
Private Gigabit network segment for the gluster servers (and the application servers that talk to them)
Geo-rep will be placed three states away in a colo center, but is currently on the same private segment.
Data:
About 6 TB of test data, the final deploy will be much larger. Most files are around 40MB in size. Files are stored in directories going 4 deep, no more than 255 files in a single directory.
All data is already compressed and encrypted with AES-256 prior to storing on the filesystem, so the comrpession feature of ZFS isn't useful to me.
The default configs for ZFS caused this to be a miserable failure. Configuring Gluster geo-replication over SSH was also a difficult task, because the documentation is wrong on many counts when compared to the version in the PPA.
With a default config, directory listings (remember, less than 255 files and folders in each directory) would take about 15 seconds to complete over the Gluster share.
By simply tweaking the following ZFS configs:
zfs set atime=off [volname]
zfs set xattr=sa [volname]
zfs set exec=off [volname]
zfs set sync=disabled [volname]
I was able to get near instantaneous directory listings and dramatically improved read performance, although I haven't bothered getting hard numbers on that.
I just wanted to pass this along. Here's my blog posts about ZFS and Gluster, which pretty much says the same thing I just did but offers more about installing ZFS:
-Tony
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
