I'm seeing less-than-stellar performance on my Gluster deployment when hosting VM images on the FUSE mount. I've seen that this topic has surfaced before, but my googling and perusing of the list archive haven't turned out very conclusive. I'm on a 2-node distribute+replicate cluster, the clients use Gluster via the FUSE mount. torbjorn at storage01:~$ sudo gluster volume info Volume Name: gluster0 Type: Distributed-Replicate Volume ID: 81bbf681-ecdb-4866-9b45-41d5d2df7b35 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: storage01.gluster.trollweb.net:/srv/gluster/brick0 Brick2: storage02.gluster.trollweb.net:/srv/gluster/brick0 Brick3: storage01.gluster.trollweb.net:/srv/gluster/brick1 Brick4: storage02.gluster.trollweb.net:/srv/gluster/brick1 The naive dd case from one client, on the dom0, looks like this: torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd if=/dev/zero of=bigfile bs=1024k count=2000 2097152000 bytes (2.1 GB) copied, 22.9161 s, 91.5 MB/s The clients see each node on a separate 1Gbps NIC, so this is pretty close to the expected transfer rate. Writing with the sync flag, from dom0, looks like so: torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd if=/dev/zero of=bigfile bs=1024k count=2000 oflag=sync 2097152000 bytes (2.1 GB) copied, 51.1271 s, 41.0 MB/s If we use a file on the gluster mount as backing for a loop device, and do a sync write: torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd if=/dev/zero of=/dev/loop1 bs=1024k count=2000 oflag=sync 2097152000 bytes (2.1 GB) copied, 56.3729 s, 37.2 MB/s The Xen instances are managed by Ganeti, using the loopback interface over a file on Gluster. Inside the Xen instance the performance is not quite what I was hoping. torbjorn at hennec:~$ sudo dd if=/dev/zero of=bigfile bs=1024k count=2000 2097152000 bytes (2.1 GB) copied, 1267.39 s, 1.7 MB/s The transfer rate is similar when using sync or direct flags with dd. Are these expected performance levels ? A couple of threads[1] talk about performance, and seem to indicate my situation isn't unique. However, I'm under the impression that other are using a similar setup with much better performance. [1]: * http://www.gluster.org/pipermail/gluster-users/2012-January/032369.html * http://www.gluster.org/pipermail/gluster-users/2012-July/033763.html -- Vennlig hilsen Torbj?rn Thorsen Utvikler / driftstekniker Trollweb Solutions AS - Professional Magento Partner www.trollweb.no Telefon dagtid: +47 51215300 Telefon kveld/helg: For kunder med Serviceavtale Bes?ksadresse: Luramyrveien 40, 4313 Sandnes Postadresse: Maurholen 57, 4316 Sandnes Husk at alle v?re standard-vilk?r alltid er gjeldende
