On Wed, Jun 27, 2012 at 03:07:21PM -0500, Nathan Stratton wrote: > >I've made a test setup like this, but unfortunately I haven't yet been able > >to get half-decent performance out of glusterfs 3.3 as a KVM backend. It > >may work better if you use local disk for the VM images, and within the VM > >mount the glusterfs volume for application data. > > What is considered half-decent? I have a 8 cluster > distribute+replicate setup and I am getting about 65MB/s and about > 1.5K IOPS. Considering that I am only using a single two disk SAS > strip in each host I think that is not bad. For a 16-disk array, your IOPS is not bad. But are you actually storing a VM image on it, and then doing lots of I/O within that VM (as opposed to mounting the volume form within the VM)? If so, can you specify your exact configuration, including OS and kernel versions? I did my tests on two quad-core/8GB nodes, 12 disks in each (md RAID10), running ubuntu 12.04, and 10GE RJ45 direct connection. The disk arrays locally perform at 350MB/s for streaming writes. But doing a dd if=/dev/zero bs=1024k within a VM, whose image was mounted on glusterfs, I was getting only 6-25MB/s. http://gluster.org/pipermail/gluster-users/2012-June/010553.html http://gluster.org/pipermail/gluster-users/2012-June/010560.html http://gluster.org/pipermail/gluster-users/2012-June/010570.html I get much better performance on locally-attached storage with O_DIRECT (kvm option "cache=none"), but have been unable to get O_DIRECT to work with glusterfs. After a kernel upgrade (to a 3.4+ kernel which supports O_DIRECT for fuse), and using the mount option direct-io-mode=enable, the VM simply wouldn't boot: http://gluster.org/pipermail/gluster-users/2012-June/010572.html http://gluster.org/pipermail/gluster-users/2012-June/010573.html Hence I'm keen to learn the recipe for good performance with glusterfs storing VM images, as if it exists, it doesn't seem to be well documented at all. Regards, Brian.