Seems now, when autoscaling is off, glusterfs running stable, at least I could not kill it with iozone. Maris Ruskulis wrote: > Thank You, for reply! As You can see from config, ping-timeout is not > set - default is asumed. Now started glusterfs with 8 threads on both > server and client (autoscaling switched off). > > Hardware: > *server1:* > lspci > 00:00.0 Host bridge: Intel Corporation E7505 Memory Controller Hub > (rev 03) > 00:00.1 Class ff00: Intel Corporation E7505/E7205 Series RAS > Controller (rev 03) > 00:01.0 PCI bridge: Intel Corporation E7505/E7205 PCI-to-AGP Bridge > (rev 03) > 00:02.0 PCI bridge: Intel Corporation E7505 Hub Interface B PCI-to-PCI > Bridge (rev 03) > 00:02.1 Class ff00: Intel Corporation E7505 Hub Interface B PCI-to-PCI > Bridge RAS Controller (rev 03) > 00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM > (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 02) > 00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM > (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 02) > 00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM > (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 02) > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 82) > 00:1f.0 ISA bridge: Intel Corporation 82801DB/DBL (ICH4/ICH4-L) LPC > Interface Bridge (rev 02) > 00:1f.1 IDE interface: Intel Corporation 82801DB (ICH4) IDE Controller > (rev 02) > 00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) > SMBus Controller (rev 02) > 02:1c.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04) > 02:1d.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge > (rev 04) > 02:1e.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04) > 02:1f.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge > (rev 04) > 03:01.0 RAID bus controller: Intel Corporation RAID Controller > 04:02.0 Ethernet controller: Intel Corporation 82540EM Gigabit > Ethernet Controller (rev 02) > 05:02.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) > 05:03.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet > Pro 100 (rev 0d) > > cat /proc/cpuinfo > processor : 0 > vendor_id : GenuineIntel > cpu family : 15 > model : 2 > model name : Intel(R) Xeon(TM) CPU 2.40GHz > stepping : 5 > cpu MHz : 2392.024 > cache size : 512 KB > physical id : 0 > siblings : 2 > core id : 0 > cpu cores : 1 > apicid : 0 > initial apicid : 0 > fdiv_bug : no > hlt_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 2 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs > bts cid xtpr > bogomips : 4784.04 > clflush size : 64 > power management: > > processor : 1 > vendor_id : GenuineIntel > cpu family : 15 > model : 2 > model name : Intel(R) Xeon(TM) CPU 2.40GHz > stepping : 5 > cpu MHz : 2392.024 > cache size : 512 KB > physical id : 0 > siblings : 2 > core id : 0 > cpu cores : 1 > apicid : 1 > initial apicid : 1 > fdiv_bug : no > hlt_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 2 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs > bts cid xtpr > bogomips : 4784.16 > clflush size : 64 > power management: > > *server2:* > lspci > 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub > (rev 0c) > 00:00.1 Class ff00: Intel Corporation E7525/E7520 Error Reporting > Registers (rev 0c) > 00:01.0 System peripheral: Intel Corporation E7520 DMA Controller (rev 0c) > 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express > Port A (rev 0c) > 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B > (rev 0c) > 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 0c) > 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 0c) > 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB > UHCI Controller #1 (rev 02) > 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB > UHCI Controller #2 (rev 02) > 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB > UHCI Controller #3 (rev 02) > 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 > EHCI Controller (rev 02) > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2) > 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC > Interface Bridge (rev 02) > 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE > Controller (rev 02) > 00:1f.3 SMBus: Intel Corporation 82801EB/ER (ICH5/ICH5R) SMBus > Controller (rev 02) > 01:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI > Bridge A (rev 09) > 01:00.1 PIC: Intel Corporation 6700/6702PXH I/OxAPIC Interrupt > Controller A (rev 09) > 01:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI > Bridge B (rev 09) > 01:00.3 PIC: Intel Corporation 6700PXH I/OxAPIC Interrupt Controller B > (rev 09) > 02:03.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 > PCI-X Fusion-MPT SAS (rev 01) > 02:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 > PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08) > 02:05.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 > PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08) > 03:01.0 I2O: LSI Logic / Symbios Logic MegaRAID (rev 01) > 05:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8050 > PCI-E ASF Gigabit Ethernet Controller (rev 18) > 07:04.0 Ethernet controller: Intel Corporation 82541GI Gigabit > Ethernet Controller (rev 05) > 07:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) > > cat /proc/cpuinfo > processor : 0 > vendor_id : GenuineIntel > cpu family : 15 > model : 4 > model name : Intel(R) Xeon(TM) CPU 2.80GHz > stepping : 1 > cpu MHz : 2792.955 > cache size : 1024 KB > physical id : 0 > siblings : 2 > core id : 0 > cpu cores : 1 > fpu : yes > fpu_exception : yes > cpuid level : 5 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > syscall nx lm constant_tsc pebs bts pni monitor ds_cpl cid cx16 xtpr > bogomips : 5590.46 > clflush size : 64 > cache_alignment : 128 > address sizes : 36 bits physical, 48 bits virtual > power management: > > processor : 1 > vendor_id : GenuineIntel > cpu family : 15 > model : 4 > model name : Intel(R) Xeon(TM) CPU 2.80GHz > stepping : 1 > cpu MHz : 2792.955 > cache size : 1024 KB > physical id : 3 > siblings : 2 > core id : 0 > cpu cores : 1 > fpu : yes > fpu_exception : yes > cpuid level : 5 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > syscall nx lm constant_tsc pebs bts pni monitor ds_cpl cid cx16 xtpr > bogomips : 5586.06 > clflush size : 64 > cache_alignment : 128 > address sizes : 36 bits physical, 48 bits virtual > power management: > > processor : 2 > vendor_id : GenuineIntel > cpu family : 15 > model : 4 > model name : Intel(R) Xeon(TM) CPU 2.80GHz > stepping : 1 > cpu MHz : 2792.955 > cache size : 1024 KB > physical id : 0 > siblings : 2 > core id : 0 > cpu cores : 1 > fpu : yes > fpu_exception : yes > cpuid level : 5 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > syscall nx lm constant_tsc pebs bts pni monitor ds_cpl cid cx16 xtpr > bogomips : 5586.02 > clflush size : 64 > cache_alignment : 128 > address sizes : 36 bits physical, 48 bits virtual > power management: > > processor : 3 > vendor_id : GenuineIntel > cpu family : 15 > model : 4 > model name : Intel(R) Xeon(TM) CPU 2.80GHz > stepping : 1 > cpu MHz : 2792.955 > cache size : 1024 KB > physical id : 3 > siblings : 2 > core id : 0 > cpu cores : 1 > fpu : yes > fpu_exception : yes > cpuid level : 5 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > syscall nx lm constant_tsc pebs bts pni monitor ds_cpl cid cx16 xtpr > bogomips : 5586.05 > clflush size : 64 > cache_alignment : 128 > address sizes : 36 bits physical, 48 bits virtual > power management: > > > > jvanwanrooy at chatventure.nl wrote: >> Hi Maris, >> >> Can you tell me something more about the hardware you use? With our >> tests yesterday we had some troubles with very high load >> inconjunction with autoscaling. You can try a fixed limit of threads. >> What are the ping-timeout settings by the way? >> >> Best Regards Jasper >> >> Jasper van Wanrooy - Chatventure BV >> Technical Manager >> T: +31 (0) 6 47 248 722 >> E: jvanwanrooy at chatventure.nl >> W: www.chatventure.nl >> >> >> ----- Original Message ----- >> From: "Maris Ruskulis" <maris at chown.lv> >> To: gluster-users at gluster.org >> Sent: Friday, 29 May, 2009 10:11:45 GMT +01:00 Amsterdam / Berlin / >> Bern / Rome / Stockholm / Vienna >> Subject: Re: Glusterfs 2.0 hangs on high load >> >> Is there way to solve this issue? >> >> Maris Ruskulis wrote: >> >> I have same issue with same config when both nodes are x64. But >> difference is that, there is no bailout messages in logs. >> >> Jasper van Wanrooy - Chatventure wrote: >> >> Hi Maris, >> >> I regret to hear that. I was also having problems with the >> stability on 32bit platforms. Possibly you should try it on a >> 64bit platform. Is that an option? >> >> Best Regards Jasper >> >> >> On 28 mei 2009, at 09:36, Maris Ruskulis wrote: >> >> Hello! >> After upgrade to version 2.0, now using 2.0.1, I'm >> experiencing problems with glusterfs stability. >> I'm running 2 node setup with cliet side afr, and >> glusterfsd also is running on same servers. Time to time >> glusterfs just hangs, i can reproduce this running iozone >> benchmarking tool. I'm using patched Fuse, but same >> result is with unpatched. >> >> ================================================================================ >> Version : glusterfs 2.0.1 built on May 27 2009 16:04:01 >> TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b >> Starting Time: 2009-05-27 16:38:20 >> Command line : /usr/sbin/glusterfsd >> --volfile=/etc/glusterfs/glusterfs-server.vol >> --pid-file=/var/run/glusterfsd.pid >> --log-file=/var/log/glusterfsd.log >> PID : 31971 >> System name : Linux >> Nodename : weeber.st-inst.lv >> Kernel Release : 2.6.28-hardened-r7 >> Hardware Identifier: i686 >> >> Given volfile: >> +------------------------------------------------------------------------------+ >> 1: # file: /etc/glusterfs/glusterfs-server.vol >> 2: volume posix >> 3: type storage/posix >> 4: option directory /home/export >> 5: end-volume >> 6: >> 7: volume locks >> 8: type features/locks >> 9: option mandatory-locks on >> 10: subvolumes posix >> 11: end-volume >> 12: >> 13: volume brick >> 14: type performance/io-threads >> 15: option autoscaling on >> 16: subvolumes locks >> 17: end-volume >> 18: >> 19: volume server >> 20: type protocol/server >> 21: option transport-type tcp >> 22: option auth.addr.brick.allow 127.0.0.1,192.168.1.* >> 23: subvolumes brick >> 24: end-volume >> >> +------------------------------------------------------------------------------+ >> [2009-05-27 16:38:20] N [glusterfsd.c:1152:main] >> glusterfs: Successfully started >> [2009-05-27 16:38:33] N >> [server-protocol.c:7035:mop_setvolume] server: accepted >> client from 192.168.1.233:1021 >> [2009-05-27 16:38:33] N >> [server-protocol.c:7035:mop_setvolume] server: accepted >> client from 192.168.1.233:1020 >> [2009-05-27 16:38:46] N >> [server-protocol.c:7035:mop_setvolume] server: accepted >> client from 192.168.1.252:1021 >> [2009-05-27 16:38:46] N >> [server-protocol.c:7035:mop_setvolume] server: accepted >> client from 192.168.1.252:1020 >> >> ================================================================================ >> Version : glusterfs 2.0.1 built on May 27 2009 16:04:01 >> TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b >> Starting Time: 2009-05-27 16:38:46 >> Command line : /usr/sbin/glusterfs -N -f >> /etc/glusterfs/glusterfs-client.vol /mnt/gluster >> PID : 32161 >> System name : Linux >> Nodename : weeber.st-inst.lv >> Kernel Release : 2.6.28-hardened-r7 >> Hardware Identifier: i686 >> >> Given volfile: >> +------------------------------------------------------------------------------+ >> 1: volume xeon >> 2: type protocol/client >> 3: option transport-type tcp >> 4: option remote-host 192.168.1.233 >> 5: option remote-subvolume brick >> 6: end-volume >> 7: >> 8: volume weeber >> 9: type protocol/client >> 10: option transport-type tcp >> 11: option remote-host 192.168.1.252 >> 12: option remote-subvolume brick >> 13: end-volume >> 14: >> 15: volume replicate >> 16: type cluster/replicate >> 17: subvolumes xeon weeber >> 18: end-volume >> 20: volume readahead >> 21: type performance/read-ahead >> 22: option page-size 128kB >> 23: option page-count 16 >> 24: option force-atime-update off >> 25: subvolumes replicate >> 26: end-volume >> 27: >> 28: volume writebehind >> 29: type performance/write-behind >> 30: option aggregate-size 1MB >> 31: option window-size 3MB >> 32: option flush-behind on >> 33: option enable-O_SYNC on >> 34: subvolumes readahead >> 35: end-volume >> 36: >> 37: volume iothreads >> 38: type performance/io-threads >> 39: option autoscaling on >> 40: subvolumes writebehind >> 41: end-volume >> 42: >> 43: >> 44: >> 45: #volume bricks >> 46: #type cluster/distribute >> 47: #option lookup-unhashed yes >> 48: #option min-free-disk 20% >> 49: # subvolumes weeber xeon >> 50: #end-volume >> >> +------------------------------------------------------------------------------+ >> [2009-05-27 16:38:46] W >> [xlator.c:555:validate_xlator_volume_options] >> writebehind: option 'window-size' is deprecated, >> preferred is 'cache-size', continuing with correction >> [2009-05-27 16:38:46] W >> [glusterfsd.c:455:_log_if_option_is_invalid] writebehind: >> option 'aggregate-size' is not recognized >> [2009-05-27 16:38:46] W >> [glusterfsd.c:455:_log_if_option_is_invalid] readahead: >> option 'page-size' is not recognized >> [2009-05-27 16:38:46] N [glusterfsd.c:1152:main] >> glusterfs: Successfully started >> [2009-05-27 16:38:46] N >> [client-protocol.c:5557:client_setvolume_cbk] xeon: >> Connected to 192.168.1.233:6996, attached to remote >> volume 'brick'. >> [2009-05-27 16:38:46] N [afr.c:2190:notify] replicate: >> Subvolume 'xeon' came back up; going online. >> [2009-05-27 16:38:46] N >> [client-protocol.c:5557:client_setvolume_cbk] xeon: >> Connected to 192.168.1.233:6996, attached to remote >> volume 'brick'. >> [2009-05-27 16:38:46] N [afr.c:2190:notify] replicate: >> Subvolume 'xeon' came back up; going online. >> [2009-05-27 16:38:46] N >> [client-protocol.c:5557:client_setvolume_cbk] weeber: >> Connected to 192.168.1.252:6996, attached to remote >> volume 'brick'. >> [2009-05-27 18:46:02] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 18:16:01. frame-timeout = 1800 >> [2009-05-27 19:16:09] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 18:46:02. frame-timeout = 1800 >> [2009-05-27 19:46:18] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame OPEN(12) frame sent = >> 2009-05-27 19:16:09. frame-timeout = 1800 >> [2009-05-27 20:16:25] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 19:46:18. frame-timeout = 1800 >> [2009-05-27 20:46:34] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 20:16:25. frame-timeout = 1800 >> [2009-05-27 21:16:41] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame OPEN(12) frame sent = >> 2009-05-27 20:46:34. frame-timeout = 1800 >> [2009-05-27 21:47:00] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 21:16:53. frame-timeout = 1800 >> [2009-05-27 22:17:07] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 21:47:00. frame-timeout = 1800 >> [2009-05-27 22:47:15] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame OPENDIR(21) frame sent = >> 2009-05-27 22:17:07. frame-timeout = 1800 >> [2009-05-27 23:17:23] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 22:47:15. frame-timeout = 1800 >> [2009-05-27 23:47:31] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame OPEN(12) frame sent = >> 2009-05-27 23:17:23. frame-timeout = 1800 >> [2009-05-28 00:17:39] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-27 23:47:32. frame-timeout = 1800 >> [2009-05-28 00:47:47] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 00:17:39. frame-timeout = 1800 >> [2009-05-28 01:17:55] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame OPENDIR(21) frame sent = >> 2009-05-28 00:47:47. frame-timeout = 1800 >> [2009-05-28 01:48:03] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 01:17:55. frame-timeout = 1800 >> [2009-05-28 02:18:11] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame OPEN(12) frame sent = >> 2009-05-28 01:48:03. frame-timeout = 1800 >> [2009-05-28 02:48:29] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 02:18:24. frame-timeout = 1800 >> [2009-05-28 03:18:37] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 02:48:29. frame-timeout = 1800 >> [2009-05-28 03:48:45] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 03:18:37. frame-timeout = 1800 >> [2009-05-28 04:18:53] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame XATTROP(40) frame sent = >> 2009-05-28 03:48:45. frame-timeout = 1800 >> [2009-05-28 04:49:01] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 04:18:53. frame-timeout = 1800 >> [2009-05-28 05:19:09] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame OPENDIR(21) frame sent = >> 2009-05-28 04:49:01. frame-timeout = 1800 >> [2009-05-28 05:49:17] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 05:19:09. frame-timeout = 1800 >> [2009-05-28 06:19:25] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 05:49:17. frame-timeout = 1800 >> [2009-05-28 06:49:33] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame XATTROP(40) frame sent = >> 2009-05-28 06:19:25. frame-timeout = 1800 >> [2009-05-28 07:19:40] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 06:49:33. frame-timeout = 1800 >> [2009-05-28 07:49:48] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 07:19:40. frame-timeout = 1800 >> [2009-05-28 08:19:56] E [client-protocol.c:292:call_bail] >> weeber: bailing out frame LOOKUP(32) frame sent = >> 2009-05-28 07:49:48. frame-timeout = 1800 >> >> <maris.vcf>_______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >> >> >> >> >> _______________________________________________ Gluster-users mailing >> list Gluster-users at gluster.org >> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >
