Rafiq -
Gluster 3.1.1 will ship shortly, in our testing performance has been
significantly improved. If you are not in production the QA builds of
Gluster 3.1.1 are here -
http://download.gluster.com/pub/gluster/glusterfs/qa-releases/. The QA
releases should NOT be run in production.
We have also found that the performance of a single EBS device can
be a limiting factor. I have found that creating lots of smaller EBS
devices then using mdadm to build a RAID 0 array can improve performance
without raising costs. I have tested with 10 EBS devices, performance
increased 4-5x for disk bound applications.
Thanks,
Craig
-->
Craig Carl
Senior Systems Engineer
Gluster
On 11/18/2010 09:14 PM, Rafiq Maniar wrote:
> Hi,
>
> I'm using Glusterfs3.1 on Ubuntu 10.04 in a dual replication setup, on
> Amazon EC2.
>
> It takes 40-50 seconds to unzip an 8MB zip file full of small files and
> directories to a gluster
> mount, in contrast to 0.8 seconds to local disk.
>
> The volume configuration was created with:
> gluster volume create volname replica 1 transport tcp server1:/shared
> server2:/shared
>
> I am mounting on the client via NFS with:
> mount -t nfs -o async,noatime,nodiratime server1:/shared /mnt/shared
>
> And also tried via Gluster native client with:
> mount -t glusterfs server1:/shared /mnt/shared
>
> I found a post here where he author talks about a similar slow unzip of the
> Linux kernel:
> http://northernmost.org/blog/improving-glusterfs-performance/
>
> I believe the 'nodelay' option was implemented in response to this, and I
> have tried using that
> in the 3 configuration files on the servers but with no improvement. I've
> also tried some other performance
> tuning tricks I found on the web.
>
> I tried it on another server that has Gluster3.0 with an NFS share but no
> replication and it completes in 3 seconds.
>
> I have similar bad performance with a simple copy of the same
> files+directories from /tmp into the gluster mount so
> its not limited to zip.
>
> Here is my /etc/glusterd/vols/shared/shared-fuse.vol.
> Bear in mind that this is 'tuned' but the out-of-the-box version is the same
> performance.
> I also tried removing all the performance translators as per someones
> suggestion in IRC.
>
> *
> *
> *volume shared-client-0*
> * type protocol/client*
> * option remote-host server1*
> * option remote-subvolume /mnt/shared*
> * option transport-type tcp*
> * option transport.socket.nodelay on*
> *end-volume*
> *
> *
> *volume shared-client-1*
> * type protocol/client*
> * option remote-host server2*
> * option remote-subvolume /mnt/shared*
> * option transport-type tcp*
> * option transport.socket.nodelay on*
> *end-volume*
> *
> *
> *volume shared-replicate-0*
> * type cluster/replicate*
> * subvolumes shared-client-0 shared-client-1*
> *end-volume*
> *
> *
> *volume shared-write-behind*
> * type performance/write-behind*
> * option cache-size 100MB*
> * option flush-behind off*
> * subvolumes shared-replicate-0*
> *end-volume*
> *
> *
> *volume shared-read-ahead*
> * type performance/read-ahead*
> * subvolumes shared-write-behind*
> *end-volume*
> *
> *
> *volume shared-io-cache*
> * type performance/io-cache*
> * option cache-size 100MB*
> * option cache-timeout 1*
> * subvolumes shared-read-ahead*
> *end-volume*
> *
> *
> *volume shared-quick-read*
> * type performance/quick-read*
> * option cache-timeout 1 # default 1 second*
> * option max-file-size 256KB # default 64Kb*
> * subvolumes shared-io-cache*
> *end-volume*
> *
> *
> *
> *
> *volume shared*
> * type debug/io-stats*
> * subvolumes shared-quick-read*
> *end-volume*
> *
> *
>
> And my /etc/glusterd/vols/shared/shared.server1.mnt-shared.vol :
>
> *volume shared-posix*
> * type storage/posix*
> * option directory /mnt/shared*
> *end-volume*
> *
> *
> *volume shared-access-control*
> * type features/access-control*
> * subvolumes shared-posix*
> *end-volume*
> *
> *
> *volume shared-locks*
> * type features/locks*
> * subvolumes shared-access-control*
> *end-volume*
> *
> *
> *volume shared-io-threads*
> * type performance/io-threads*
> * option thread-count 16*
> * subvolumes shared-locks*
> *end-volume*
> *
> *
> *volume /mnt/shared*
> * type debug/io-stats*
> * subvolumes shared-io-threads*
> *end-volume*
> *
> *
> *volume shared-server*
> * type protocol/server*
> * option transport-type tcp*
> * option auth.addr./mnt/shared.allow **
> * option transport.socket.nodelay on*
> * subvolumes /mnt/shared*
> *end-volume*
>
>
> Here's the output of nfsstat on the client:
> *Client rpc stats:*
> *calls retrans authrefrsh*
> *1652499 231 124*
> *
> *
> *Client nfs v3:*
> *null getattr setattr lookup access readlink*
> *0 0% 744498 45% 32762 1% 490843 29% 235276 14% 37
> 0%*
> *read write create mkdir symlink mknod*
> *52085 3% 21940 1% 14452 0% 948 0% 1 0% 0
> 0%*
> *remove rmdir rename link readdir
> readdirplus*
> *10961 0% 562 0% 19 0% 0 0% 135 0% 32623
> 1%*
> *fsstat fsinfo pathconf commit*
> *140 0% 46 0% 23 0% 15126 0%*
>
>
> Anyone got any ideas on improving the performance of this?
>
> Thanks,
> Rafiq
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
