Are there any other things I can try here or have I dug up a bug that is being looked at still? We'd love to roll out glusterfs into production use, but the performance with small files is unacceptable. Ben On Jun 5, 2009, at 9:02 AM, Benjamin Krein wrote: > > On Jun 5, 2009, at 2:23 AM, Pavan Vilas Sondur wrote: > >> Hi Benjamin, >> Can you provide us with the following information to narrow down >> the issue. >> 1. The network configuration of the GlusterFS deployment and the >> bandwidth that is available >> for GlusterFS to use > > All these servers (servers/clients) are on a full-duplex gigabit LAN > with gigabit NICs. As I've noted in previous emails, I can easily > utilize the bandwidth with scp & rsync. As you'll see below, > glusterfs seems to do very well with large files as well. > >> 2. Does performance also get affected the same way with large files >> too? Or is it with just small files >> as you have mentioned? Let us know the performance of GlusterFS >> with large files. > > Here are a bunch of tests I did with various configurations & > copying large files: > > * Single server - cfs1 - large files (~480MB each) > root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v > webform_cache.tar /mnt/large_file_\$i.tar; done;" > `webform_cache.tar' -> `/mnt/large_file_1.tar' > `webform_cache.tar' -> `/mnt/large_file_2.tar' > `webform_cache.tar' -> `/mnt/large_file_3.tar' > `webform_cache.tar' -> `/mnt/large_file_4.tar' > `webform_cache.tar' -> `/mnt/large_file_5.tar' > > real 0m23.726s > user 0m0.128s > sys 0m4.972s > > # Interface RX Rate RX # TX > Rate TX # > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ??????????????????????? > cfs1 (source: local) > 0 eth1 91.39MiB 63911 > 1.98MiB 27326 > > * Two servers w/ AFR only - large files (~480MB each) > root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v > webform_cache.tar /mnt/large_file_\$i.tar; done;" > `webform_cache.tar' -> `/mnt/large_file_1.tar' > `webform_cache.tar' -> `/mnt/large_file_2.tar' > `webform_cache.tar' -> `/mnt/large_file_3.tar' > `webform_cache.tar' -> `/mnt/large_file_4.tar' > `webform_cache.tar' -> `/mnt/large_file_5.tar' > > real 0m43.354s > user 0m0.100s > sys 0m3.044s > > # Interface RX Rate RX # TX > Rate TX # > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ??????????????????????? > cfs1 (source: local) > 0 eth1 57.73MiB 39931 > 910.95KiB 10733 > > * Two servers w/DHT+AFR - large files (~480MB each) > root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v > webform_cache.tar /mnt/large_file_\$i.tar; done;" > `webform_cache.tar' -> `/mnt/large_file_1.tar' > `webform_cache.tar' -> `/mnt/large_file_2.tar' > `webform_cache.tar' -> `/mnt/large_file_3.tar' > `webform_cache.tar' -> `/mnt/large_file_4.tar' > `webform_cache.tar' -> `/mnt/large_file_5.tar' > > real 0m43.294s > user 0m0.100s > sys 0m3.356s > > # Interface RX Rate RX # TX > Rate TX # > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ??????????????????????? > cfs1 (source: local) > 0 eth1 58.15MiB 40224 > 1.52MiB 20174 > # Interface RX Rate RX # TX > Rate TX # > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ??????????????????????? > cfs2 (source: local) > 0 eth1 55.99MiB 38684 > 755.51KiB 9521 > > * Two servers DHT *only* - large files (~480MB each) - NOTE: only > cfs1 was ever populated, isn't DHT supposed to distribute the files?: > root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v > webform_cache.tar /mnt/\$i/large_file.tar; done;" > `webform_cache.tar' -> `/mnt/1/large_file.tar' > `webform_cache.tar' -> `/mnt/2/large_file.tar' > `webform_cache.tar' -> `/mnt/3/large_file.tar' > `webform_cache.tar' -> `/mnt/4/large_file.tar' > `webform_cache.tar' -> `/mnt/5/large_file.tar' > > real 0m40.062s > user 0m0.204s > sys 0m3.500s > > # Interface RX Rate RX # TX > Rate TX # > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ??????????????????????? > cfs1 (source: local) > 0 eth1 112.96MiB 78190 > 1.66MiB 21994 > # Interface RX Rate RX # TX > Rate TX # > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ? > ??????????????????????? > cfs2 (source: local) > 0 eth1 1019.00B 7 > 83.00B 0 > > root at cfs1|/home/clusterfs/webform/cache2|# find -ls > 294926 8 drwxrwxr-x 7 www-data www-data 4096 Jun 5 08:52 . > 294927 8 drwxr-xr-x 2 root root 4096 Jun 5 08:53 ./1 > 294933 489716 -rw-r--r-- 1 root root 500971520 Jun 5 > 08:54 ./1/large_file.tar > 294931 8 drwxr-xr-x 2 root root 4096 Jun 5 08:53 ./5 > 294932 489716 -rw-r--r-- 1 root root 500971520 Jun 5 > 08:54 ./5/large_file.tar > 294930 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./4 > 294936 489716 -rw-r--r-- 1 root root 500971520 Jun 5 > 08:54 ./4/large_file.tar > 294928 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./2 > 294934 489716 -rw-r--r-- 1 root root 500971520 Jun 5 > 08:54 ./2/large_file.tar > 294929 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./3 > 294935 489716 -rw-r--r-- 1 root root 500971520 Jun 5 > 08:54 ./3/large_file.tar > > > root at cfs2|/home/clusterfs/webform/cache2|# find -ls > 3547150 8 drwxrwxr-x 7 www-data www-data 4096 Jun 5 08:52 . > 3547153 8 drwxr-xr-x 2 root root 4096 Jun 5 > 08:52 ./3 > 3547155 8 drwxr-xr-x 2 root root 4096 Jun 5 > 08:52 ./5 > 3547154 8 drwxr-xr-x 2 root root 4096 Jun 5 > 08:52 ./4 > 3547152 8 drwxr-xr-x 2 root root 4096 Jun 5 > 08:52 ./2 > 3547151 8 drwxr-xr-x 2 root root 4096 Jun 5 > 08:52 ./1 > >> >> There haven't really been an issue with using large number of small >> files as such in the past. Nevertheless, >> we'll look into this once you give us the above details. >> > > I don't feel that the tests I'm performing are all that abnormal or > out of the ordinary, so I'm surprised that I'm the only one having > the problems. As you can see from the large file results above, the > problem is clearly related to small files only. > > Thanks for your continued interest in resolving this! > > Ben > >> >> On 04/06/09 16:21 -0400, Benjamin Krein wrote: >>> Here are some more details with different configs: >>> >>> * Only AFR between cfs1 & cfs2: >>> root at dev1# time cp -rp * /mnt/ >>> >>> real 16m45.995s >>> user 0m1.104s >>> sys 0m5.528s >>> >>> * Single server - cfs1: >>> root at dev1# time cp -rp * /mnt/ >>> >>> real 10m33.967s >>> user 0m0.764s >>> sys 0m5.516s >>> >>> * Stats via bmon on cfs1 during above copy: >>> # Interface RX Rate RX # TX Rate >>> TX # >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ? >>> ?????????????????????? >>> ? >>> ? >>> ?????????????????????? >>> cfs1 (source: local) >>> 0 eth1 951.25KiB 1892 254.00KiB >>> 1633 >>> >>> It gets progressively better, but that's still a *long* way from >>> <2 min >>> times with scp & <1 min times with rsync! And, I have no >>> redundancy or >>> distributed hash whatsoever. >>> >>> * Client config for the last test: >>> ----- >>> # Webform Flat-File Cache Volume client configuration >>> >>> volume srv1 >>> type protocol/client >>> option transport-type tcp >>> option remote-host cfs1 >>> option remote-subvolume webform_cache_brick >>> end-volume >>> >>> volume writebehind >>> type performance/write-behind >>> option cache-size 4mb >>> option flush-behind on >>> subvolumes srv1 >>> end-volume >>> >>> volume cache >>> type performance/io-cache >>> option cache-size 512mb >>> subvolumes writebehind >>> end-volume >>> ----- >>> >>> Ben >>> >>> On Jun 3, 2009, at 4:33 PM, Vahri? Muhtaryan wrote: >>> >>>> For better understanding issue did you try 4 servers DHT only or 2 >>>> servers >>>> DHT only or two servers replication only for find out real problem >>>> maybe >>>> replication or dht could have a bug ? >>>> >>>> -----Original Message----- >>>> From: gluster-users-bounces at gluster.org >>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Benjamin >>>> Krein >>>> Sent: Wednesday, June 03, 2009 11:00 PM >>>> To: Jasper van Wanrooy - Chatventure >>>> Cc: gluster-users at gluster.org >>>> Subject: Re: Horrible performance with small files >>>> (DHT/AFR) >>>> >>>> The current boxes I'm using for testing are as follows: >>>> >>>> * 2x dual-core Opteron ~2GHz (x86_64) >>>> * 4GB RAM >>>> * 4x 7200 RPM 73GB SATA - RAID1+0 w/3ware hardware controllers >>>> >>>> The server storage directories live in /home/clusterfs where / >>>> home is >>>> an ext3 partition mounted with noatime. >>>> >>>> These servers are not virtualized. They are running Ubuntu 8.04 >>>> LTS >>>> Server x86_64. >>>> >>>> The files I'm copying are all <2k javascript files (plain text) >>>> stored >>>> in 100 hash directories in each of 3 parent directories: >>>> >>>> /home/clusterfs/ >>>> + parentdir1/ >>>> | + 00/ >>>> | | ... >>>> | + 99/ >>>> + parentdir1/ >>>> | + 00/ >>>> | | ... >>>> | + 99/ >>>> + parentdir1/ >>>> + 00/ >>>> | ... >>>> + 99/ >>>> >>>> There are ~10k of these <2k javascript files distributed throughout >>>> the above directory structure totaling approximately 570MB. My >>>> tests >>>> have been copying that entire directory structure from a client >>>> machine into the glusterfs mountpoint on the client. >>>> >>>> Observing IO on both the client box & all the server boxes via >>>> iostat >>>> shows that the disks are doing *very* little work. Observing the >>>> CPU/ >>>> memory load with top or htop shows that none of the boxes are CPU >>>> or >>>> memory bound. Observing the bandwidth in/out of the network >>>> interface >>>> shows <1MB/s throughput (we have a fully gigabit LAN!) which >>>> usually >>>> drops down to <150KB/s during the copy. >>>> >>>> scp'ing the same directory structure from the same client to one of >>>> the same servers will work at ~40-50MB/s sustained as a comparison. >>>> Here is the results of copying the same directory structure using >>>> rsync to the same partition: >>>> >>>> # time rsync -ap * benk at cfs1:~/cache/ >>>> benk at cfs1's password: >>>> >>>> real 0m23.566s >>>> user 0m8.433s >>>> sys 0m4.580s >>>> >>>> Ben >>>> >>>> On Jun 3, 2009, at 3:16 PM, Jasper van Wanrooy - Chatventure wrote: >>>> >>>>> Hi Benjamin, >>>>> >>>>> That's not good news. What kind of hardware do you use? Is it >>>>> virtualised? Or do you use real boxes? >>>>> What kind of files are you copying in your test? What >>>>> performance do >>>>> you have when copying it to a local dir? >>>>> >>>>> Best regards Jasper >>>>> >>>>> ----- Original Message ----- >>>>> From: "Benjamin Krein" <superbenk at superk.org> >>>>> To: "Jasper van Wanrooy - Chatventure" >>>>> <jvanwanrooy at chatventure.nl> >>>>> Cc: "Vijay Bellur" <vijay at gluster.com>, gluster-users at gluster.org >>>>> Sent: Wednesday, 3 June, 2009 19:23:51 GMT +01:00 Amsterdam / >>>>> Berlin / Bern / Rome / Stockholm / Vienna >>>>> Subject: Re: Horrible performance with small files >>>>> (DHT/AFR) >>>>> >>>>> I reduced my config to only 2 servers (had to donate 2 of the 4 to >>>>> another project). I now have a single server using DHT (for >>>>> future >>>>> scaling) and AFR to a mirrored server. Copy times are much >>>>> better, >>>>> but still pretty horrible: >>>>> >>>>> # time cp -rp * /mnt/ >>>>> >>>>> real 21m11.505s >>>>> user 0m1.000s >>>>> sys 0m6.416s >>>>> >>>>> Ben >>>>> >>>>> On Jun 3, 2009, at 3:13 AM, Jasper van Wanrooy - Chatventure >>>>> wrote: >>>>> >>>>>> Hi Benjamin, >>>>>> >>>>>> Did you also try with a lower thread-count. Actually I'm using 3 >>>>>> threads. >>>>>> >>>>>> Best Regards Jasper >>>>>> >>>>>> >>>>>> On 2 jun 2009, at 18:25, Benjamin Krein wrote: >>>>>> >>>>>>> I do not see any difference with autoscaling removed. Current >>>>>>> server config: >>>>>>> >>>>>>> # webform flat-file cache >>>>>>> >>>>>>> volume webform_cache >>>>>>> type storage/posix >>>>>>> option directory /home/clusterfs/webform/cache >>>>>>> end-volume >>>>>>> >>>>>>> volume webform_cache_locks >>>>>>> type features/locks >>>>>>> subvolumes webform_cache >>>>>>> end-volume >>>>>>> >>>>>>> volume webform_cache_brick >>>>>>> type performance/io-threads >>>>>>> option thread-count 32 >>>>>>> subvolumes webform_cache_locks >>>>>>> end-volume >>>>>>> >>>>>>> <<snip>> >>>>>>> >>>>>>> # GlusterFS Server >>>>>>> volume server >>>>>>> type protocol/server >>>>>>> option transport-type tcp >>>>>>> subvolumes dns_public_brick dns_private_brick >>>>>>> webform_usage_brick >>>>>>> webform_cache_brick wordpress_uploads_brick subs_exports_brick >>>>>>> option auth.addr.dns_public_brick.allow 10.1.1.* >>>>>>> option auth.addr.dns_private_brick.allow 10.1.1.* >>>>>>> option auth.addr.webform_usage_brick.allow 10.1.1.* >>>>>>> option auth.addr.webform_cache_brick.allow 10.1.1.* >>>>>>> option auth.addr.wordpress_uploads_brick.allow 10.1.1.* >>>>>>> option auth.addr.subs_exports_brick.allow 10.1.1.* >>>>>>> end-volume >>>>>>> >>>>>>> # time cp -rp * /mnt/ >>>>>>> >>>>>>> real 70m13.672s >>>>>>> user 0m1.168s >>>>>>> sys 0m8.377s >>>>>>> >>>>>>> NOTE: the above test was also done during peak hours when the >>>>>>> LAN/ >>>>>>> dev server were in use which would cause some of the extra time. >>>>>>> This is still WAY too much, though. >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>> >>>>>>> On Jun 1, 2009, at 1:40 PM, Vijay Bellur wrote: >>>>>>> >>>>>>>> Hi Benjamin, >>>>>>>> >>>>>>>> Could you please try by turning autoscaling off? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Vijay >>>>>>>> >>>>>>>> Benjamin Krein wrote: >>>>>>>>> I'm seeing extremely poor performance writing small files to a >>>>>>>>> glusterfs DHT/AFR mount point. Here are the stats I'm seeing: >>>>>>>>> >>>>>>>>> * Number of files: >>>>>>>>> root at dev1|/home/aweber/cache|# find |wc -l >>>>>>>>> 102440 >>>>>>>>> >>>>>>>>> * Average file size (bytes): >>>>>>>>> root at dev1|/home/aweber/cache|# ls -lR | awk '{sum += $5; n++;} >>>>>>>>> END {print sum/n;}' >>>>>>>>> 4776.47 >>>>>>>>> >>>>>>>>> * Using scp: >>>>>>>>> root at dev1|/home/aweber/cache|# time scp -rp * benk at cfs1:~/ >>>>>>>>> cache/ >>>>>>>>> >>>>>>>>> real 1m38.726s >>>>>>>>> user 0m12.173s >>>>>>>>> sys 0m12.141s >>>>>>>>> >>>>>>>>> * Using cp to glusterfs mount point: >>>>>>>>> root at dev1|/home/aweber/cache|# time cp -rp * /mnt >>>>>>>>> >>>>>>>>> real 30m59.101s >>>>>>>>> user 0m1.296s >>>>>>>>> sys 0m5.820s >>>>>>>>> >>>>>>>>> Here is my configuration (currently, single client writing >>>>>>>>> to 4 >>>>>>>>> servers (2 DHT servers doing AFR): >>>>>>>>> >>>>>>>>> SERVER: >>>>>>>>> >>>>>>>>> # webform flat-file cache >>>>>>>>> >>>>>>>>> volume webform_cache >>>>>>>>> type storage/posix >>>>>>>>> option directory /home/clusterfs/webform/cache >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume webform_cache_locks >>>>>>>>> type features/locks >>>>>>>>> subvolumes webform_cache >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume webform_cache_brick >>>>>>>>> type performance/io-threads >>>>>>>>> option thread-count 32 >>>>>>>>> option max-threads 128 >>>>>>>>> option autoscaling on >>>>>>>>> subvolumes webform_cache_locks >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> <<snip>> >>>>>>>>> >>>>>>>>> # GlusterFS Server >>>>>>>>> volume server >>>>>>>>> type protocol/server >>>>>>>>> option transport-type tcp >>>>>>>>> subvolumes dns_public_brick dns_private_brick >>>>>>>>> webform_usage_brick >>>>>>>>> webform_cache_brick wordpress_uploads_brick subs_exports_brick >>>>>>>>> option auth.addr.dns_public_brick.allow 10.1.1.* >>>>>>>>> option auth.addr.dns_private_brick.allow 10.1.1.* >>>>>>>>> option auth.addr.webform_usage_brick.allow 10.1.1.* >>>>>>>>> option auth.addr.webform_cache_brick.allow 10.1.1.* >>>>>>>>> option auth.addr.wordpress_uploads_brick.allow 10.1.1.* >>>>>>>>> option auth.addr.subs_exports_brick.allow 10.1.1.* >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> CLIENT: >>>>>>>>> >>>>>>>>> # Webform Flat-File Cache Volume client configuration >>>>>>>>> >>>>>>>>> volume srv1 >>>>>>>>> type protocol/client >>>>>>>>> option transport-type tcp >>>>>>>>> option remote-host cfs1 >>>>>>>>> option remote-subvolume webform_cache_brick >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume srv2 >>>>>>>>> type protocol/client >>>>>>>>> option transport-type tcp >>>>>>>>> option remote-host cfs2 >>>>>>>>> option remote-subvolume webform_cache_brick >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume srv3 >>>>>>>>> type protocol/client >>>>>>>>> option transport-type tcp >>>>>>>>> option remote-host cfs3 >>>>>>>>> option remote-subvolume webform_cache_brick >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume srv4 >>>>>>>>> type protocol/client >>>>>>>>> option transport-type tcp >>>>>>>>> option remote-host cfs4 >>>>>>>>> option remote-subvolume webform_cache_brick >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume afr1 >>>>>>>>> type cluster/afr >>>>>>>>> subvolumes srv1 srv3 >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume afr2 >>>>>>>>> type cluster/afr >>>>>>>>> subvolumes srv2 srv4 >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume dist >>>>>>>>> type cluster/distribute >>>>>>>>> subvolumes afr1 afr2 >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume writebehind >>>>>>>>> type performance/write-behind >>>>>>>>> option cache-size 4mb >>>>>>>>> option flush-behind on >>>>>>>>> subvolumes dist >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> volume cache >>>>>>>>> type performance/io-cache >>>>>>>>> option cache-size 512mb >>>>>>>>> subvolumes writebehind >>>>>>>>> end-volume >>>>>>>>> >>>>>>>>> Benjamin Krein >>>>>>>>> www.superk.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
