Hi Pavan, Thanks for the reply - my comments inline below Regards, Thomas -----Original Message----- From: Pavan T C [mailto:tcp at gluster.com] Sent: Wednesday, 14 September 2011 9:19 PM To: Thomas Jackson Cc: gluster-users at gluster.org Subject: Re: Slow performance - 4 hosts, 10 gigabit ethernet, Gluster 3.2.3 > On Friday 09 September 2011 10:30 AM, Thomas Jackson wrote: >> Hi everyone, > Hello Thomas, > Try the following: > 1. In the fuse volume file, try: > Under write-behind: > "option cache-size 16MB" > Under read-ahead: > "option page-count 16" > Under io-cache: > "option cache-size=64MB" TJ: Results here are not pretty! root at my-host:~# dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=10000 10485760000 bytes (10 GB) copied, 107.888 s, 97.2 MB/s > 2. Did you get 9Gbits/Sec with iperf with a single thread or multiple threads? TJ: Single thread > 3. Can you give me the output of: > sysctl -a | egrep 'rmem|wmem' TJ: root at my-host:~# sysctl -a | egrep 'rmem|wmem' error: permission denied on key 'vm.compact_memory' vm.lowmem_reserve_ratio = 256 256 32 error: permission denied on key 'net.ipv4.route.flush' net.core.wmem_max = 131071 net.core.rmem_max = 131071 net.core.wmem_default = 126976 net.core.rmem_default = 126976 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 error: permission denied on key 'net.ipv6.route.flush' > 4. If it is not a problem for you, can you please create a pure distribute setup (instead of distributed-replicate) and then report the numbers? TJ: I've been able to do this with 2 hosts, while I was at it I also tested a pure replica and pure stripe setup for comparison Distribute = 313 MB/sec Replica = 166 MB/sec Stripe = 529 MB/sec > 5. What is the inode size with which you formatted you XFS filesystem ? > This last point might not be related to your throughput problem, but if you are planning to use this setup for a large number of files, > you might be better off using an inode size of 512 instead of the default 256 bytes. To do that, your mkfs command should be: > mkfs -t xfs -i size=512 /dev/<disk device> TJ: This is destined for use with VM images, probably a maximum of 200 files total. That said, I have tried with a bigger inode size and also with ext4 with very similar results each time In a totally bizarre turn of events, turning on port bonding (each host has 2x 10gig storage ports) in ACTIVE/BACKUP mode has increased the speed a fair bit dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000 41943040000 bytes (42 GB) copied, 176.459 s, 238 MB/s I have noticed that the inodes are getting very frequently locked/unlocked by the afs from some brief debugging, not sure if that is related > Pavan >> I am seeing slower-than-expected performance in Gluster 3.2.3 between >> 4 hosts with 10 gigabit eth between them all. Each host has 4x 300GB >> SAS 15K drives in RAID10, 6-core Xeon E5645 @ 2.40GHz and 24GB RAM >> running Ubuntu >> 10.04 64-bit (I have also tested with Scientific Linux 6.1 and Debian >> Squeeze - same results on those as well). All of the hosts mount the >> volume using the FUSE module. The base filesystem on all of the nodes >> is XFS, however tests with ext4 have yielded similar results. >> >> Command used to create the volume: >> gluster volume create cluster-volume replica 2 transport tcp >> node01:/mnt/local-store/ node02:/mnt/local-store/ >> node03:/mnt/local-store/ node04:/mnt/local-store/ >> >> Command used to mount the Gluster volume on each node: >> mount -t glusterfs localhost:/cluster-volume /mnt/cluster-volume >> >> Creating a 40GB file onto a node's local storage (ie no Gluster >> involvement): >> dd if=/dev/zero of=/mnt/local-store/test.file bs=1M count=40000 >> 41943040000 bytes (42 GB) copied, 92.9264 s, 451 MB/s >> >> Getting the same file off the node's local storage: >> dd if=/mnt/local-store/test.file of=/dev/null >> 41943040000 bytes (42 GB) copied, 81.858 s, 512 MB/s >> >> 40GB file onto the Gluster storage: >> dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000 >> 41943040000 bytes (42 GB) copied, 226.934 s, 185 MB/s >> >> Getting the same file off the Gluster storage >> dd if=/mnt/cluster-volume/test.file of=/dev/null >> 41943040000 bytes (42 GB) copied, 661.561 s, 63.4 MB/s >> >> I have also tried using Gluster 3.1, with similar results. >> >> According to the Gluster docs, I should be seeing roughly the lesser >> of the drive speed and the network speed. The network is able to push >> 0.9GB/sec according to iperf so that definitely isn't a limiting >> factor here, and each array is able to do 400-500MB/sec as per above >> benchmarks. I've tried with/without jumbo frames as well, which doesn't make any major difference. >> >> The glusterfs process is using 120% CPU according to top, and >> glusterfsd is sitting at about 90%. >> >> Any ideas / tips of where to start for speeding this config up? >> >> Thanks, >> >> Thomas >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Message protected by MailGuard: e-mail anti-virus, anti-spam and content filtering. http://www.mailguard.com.au Click here to report this message as spam: https://login.mailguard.com.au/report/1D7vGPSlkV/1bIPKtdYfJHjPGDWxiqyFd/2.76 6
