Hi Ramiro ideas off the top of my head: Get rid of performance/quick-read - it has a memory leak bug due to be fixed in gluster v3.0.5 If the files are going to be accessed by a program (which doesn't list the directories often) rather than a user (who might) then you can get rid of performance/stat-prefetch too. In the long term - will this cluster be mostly used for serving files (ie. read-mostly) or will you be creating files as often as reading them? If mostly-read then get rid of performance/write-behind. Also the calculated option cache-size only uses 20% of your ram for the io-cache. Hard code it to a value you know works best. See also my write-up http://www.sirgroane.net/2010/03/tuning-glusterfs-for-apache-on-ec2 perhaps. As you have physical disks then I'm guessing performance/read-ahead should be good for you. Does the genfiles.sh script allow you to simulate multiple processes - if not then you're not seeing the full benefit of your 6 back-end stores... Cheers, Ian On 26/03/2010 17:04, Ramiro Magallanes wrote: > Hello there! > > Im working on a 6-nodes cluster, with SuperMicro new hardware. > The cluster have to store a millons of JPG's about (200k-4MB),and little > text files. > > Each node is : > > -Single Xeon(R) CPU E5405 @ 2.00GHz (4 cores) > -4 GB RAM > -64 bits Distro-based (Debian Lenny) > -3ware 9650 sataII-raid, with 1 logical drive in raid 5 mode, the unit > with 3 sata hardisk of 2TB wdc with 64MB of cache each one. > -Xfs filesystem on each logical unit. > > When i run the "genfiles.sh" test on each node in local (in the raid-5 > unit) mode i've have the follow results: > > -3143 files created in 60 seconds. > > and if i comment the "sync" line in the script: > > -8947 files created in 60 seconds. > > Now , with Gluster mounted (22TB) i run the test and the results are: > > -1370 files created in 60 seconds. > > Now, I'm running the cluster with standard distributed configuration, > and i was making significant number of change in the test process , but > i obtain the same number of wroted files all the time. > Never more than 1400 files created, and 170mbits of network load (top). > > The switching layer is gigabit (obviusly) , and there's no high > resources being used , all is normal. > > I'm using the 3.0.3 version of Gluster. > > Here is my configuration file (only the last part of the file): > > ############################################################################## > volume distribute > type cluster/distribute > subvolumes 172.17.15.1-1 172.17.15.2-1 172.17.15.3-1 > 172.17.15.4-1 172.17.15.5-1 172.17.15.6-1 > end-volume > > volume writebehind > type performance/write-behind > option cache-size 1MB > option flush-behind on > subvolumes distribute > end-volume > > volume readahead > type performance/read-ahead > option page-count 4 > subvolumes writebehind > end-volume > > volume iocache > type performance/io-cache > option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print > $2 * 0.2 / 1024}' | cut -f1 -d.`MB > > option cache-timeout 1 > subvolumes readahead > end-volume > > volume iothreads > type performance/io-threads > option thread-count 32 # default is 16 > subvolumes distribute > end-volume > > volume quickread > type performance/quick-read > option cache-timeout 1 > option max-file-size 128kB > subvolumes iocache > end-volume > > volume statprefetch > type performance/stat-prefetch > subvolumes quickread > end-volume > ############################################################################## > > Any idea or suggestion to make the performance goes up? > Thanks everyone! > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > -- www.ContactClean.com Making changing email address as easy as clicking a mouse. Helping you keep in touch.
