Hi Joe. Your remarks are always useful and informative - thanks. As for our slow network throughput - what I didn't put in our configuration is that our servers are on 10GBe, but all of our clients are on 1Gbe because our core network can't (yet) handle the load of 10GBe clients - just to fill in that data point. I find your remark about the stability of 3.1.3 reassuring - considering the painful struggle to get there from the 3.0.4 versions, and the stability issues that I noted in the list over the course of my migration. Your problems with 3.1.4 and 3.2 are enough reason to not do anymore upgrades to the production systems yet. As a rule of thumb, I never implement X.0 releases into production anyhow - even from Redhat ... I have the arrows still sticking out of my back from doing so in the past :-) James Burnash, Unix Engineering -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Joe Landman Sent: Wednesday, May 11, 2011 10:36 AM To: gluster-users at gluster.org Subject: Re: [SPAM?] Storage Design Overview On 05/11/2011 10:22 AM, Burnash, James wrote: Standard disclaimers apply ... we build really fast storage systems and storage clusters and have a financial interest in these things, so take what we say in this context. > Answers inline below as well J > > Hope this helps. > > James Burnash, Unix Engineering > > *From:*gluster-users-bounces at gluster.org > [mailto:gluster-users-bounces at gluster.org] *On Behalf Of *Nyamul Hassan > *Sent:* Wednesday, May 11, 2011 10:04 AM > *To:* gluster-users at gluster.org > *Subject:* Re: [SPAM?] Storage Design Overview > > Thank you for the prompt and insightful answer, James. My remarks are > inline. > > 1.Can we mount a GlusterFS on a client and expect it to provide > sustained throughput near wirespeed? <No> > > In your scenario, what were the maximum read speeds that you observed? > > Read (using dd) approximately 60MB/sec to 100MB/sec. Depends upon many things in a long chain ... network performance, local stack performance, remote disk performance, etc. Our experience has been that the cause of a majority of the lower performing situations we have observed in self-designed systems, has been a significant (often severe and designed in) bottleneck, that actively prevents users from achieving anything more than moderate speed. We have measured up to 700 MB/s for simple dd's over an SDR Infiniband network using gluster 3.1.3, and about 500 MB/s over 10GbE. It is achievable, but you have to start with a good design. Good designs aren't buzzword enabled ... there are methods to the madness as it were. This is what we provide to our customer base. > > 3.Does it put extra pressure on the client?<What do you mean by > pressure? My clients (HP ProLiant DL360 G5 Quad Core with 32GB RAM) > show up to 2GB of memory usage when the native Gluster client is > used for mounts ? but that is dependent on what you set the client > cache max for ? in my case, 2GB. CPU utilization is usually > negligible in my systems, network bandwidth utilization and I/O > throughput ? depend on what the files sizes and access patterns look > like> Heavy IO will fill up work queue slots in the kernel. This is true of every file system. > > Thx for the insight. Can you describe your current deployment a bit > more, like configs of the storage nodes, and the client nodes, and what > type of application you are using it for? Don't want to be too > intrusive, just to get an idea on what others are doing. > > All on Gluster 3.1.3 We are also at 3.1.3 in the lab after experiencing problems with 3.1.4 and 3.2.0. Have a few bugs filed. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: landman at scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
