Michael, Thanks for your suggestion. I had the same intuition as you, but then I used "iptraf" and saw no eth0 traffic associated with I/O on the Gluster volume. (the tool doesn't show the ib0 interface, unfortunately). node01 and node02 are entered into /etc/hosts, they resolve to the ipoib addresses and are pingable. I will try increasing the number of threads and applying the patch Bryan suggested. Thanks, Adrian ________________________________________ From: Michael Mayer [michael at mayer.cx] Sent: 21 April 2012 10:21 To: Bryan Whitehead Cc: Ionescu, A.; gluster-users at gluster.org Subject: Re: Performance issues with striped volume over Infiniband Adrian, I think you should really try to restart from scratch, i.e. - setup ipoib with corresponding entries in DNS or /etc/hosts (make sure the IB hosts are pingable) - gluster peer probe on the ipoib hostnames - recreate volume using ipoib hostnames for the bricks For transport you can still use tcp (I usually set tcp,rdma to be flexible later on) I started naively myself a couple of weeks ago and did exactly the same than you assuming that IB transport would be autodetected. In fact, the 100 MB/s you are seeing indicated very strongly that any communication is still over Gigabit Ethernet (Which has a theoretical peak performance of 125 MB/s but given switch quality and protocol overhead you will reach more like 100-110 MB/s in reality) Michael. On 04/20/2012 07:42 PM, Bryan Whitehead wrote: > Max out the number of IO threads and apply a patch to make gluster > more agressive about spawning threads as in this thread: > > http://gluster.org/pipermail/gluster-users/2012-February/009590.html > > (the above thread is actually pretty good for getting performance out > of gluster with infiniband (I use IP over IB myself - could never get > the native rDMA to be fast enough). > > On Fri, Apr 20, 2012 at 3:27 AM, Ionescu, A.<a.ionescu at student.vu.nl> wrote: >> I tried this, but the performance is still the same (about ~100MB/s). >> >> A strange thing I noticed is that the node on which I run the benchmarking tool reports ~350 MB/s traffic on the loopback interface. >> >> Does anyone have any other suggestions? >> >> Thanks, >> Adrian >> ________________________________________ >> From: gluster-users-bounces at gluster.org [gluster-users-bounces at gluster.org] on behalf of Sabuj Pattanayek [sabujp at gmail.com] >> Sent: 19 April 2012 22:32 >> To: gluster-users at gluster.org >> Subject: Re: Performance issues with striped volume over Infiniband >> >> On Thu, Apr 19, 2012 at 2:00 PM, Ionescu, A.<a.ionescu at student.vu.nl> wrote: >>> Thanks for your answer, Sabuj. However, I am not sure I understand what you mean by trying with ipoib. >>> Do you mean specifying "transport tcp" and using the ipoib ips/hostnames for the bricks? >>> If you manage to find a way to overcome this performance limitation, please share with us your solution. >> Yes. >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
