Hi Bryan, I am copying the "gluster volume info all" output below. Sorry about the confusion with IPoIB and RDMA. I am using RDMA on the main-volume (mount -t glusterfs cfdstorage01-ib:main-volume.rdma /cfd/data) and IPoIB on backup volume (mount -t glusterfs cfdstorage01-ib:backup-volume /cfd/data) as gluster will not let me do two RDMA mounts at once. The confusion probably arose that I was initially setting up a cluster with the Gigabit Ethernet associated host names as peers. When I used rdma in this setup I could not get much more than 100 MB/s. Only when I replaced the GbE host names with the IPoIB ones, performance increased to about 400...500 MB/s (single thread dd with 1 MB block size). With the RDMA I am getting up to about 900 MB/s. Hope that answers your questions ? Michael. Volume Name: backup-volume Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp,rdma Bricks: Brick1: cfdstorage01-ib:/export/backup/0 Brick2: cfdstorage02-ib:/export/backup/0 Brick3: cfdstorage03-ib:/export/backup/0 Brick4: cfdstorage04-ib:/export/backup/0 Brick5: cfdstorage01-ib:/export/backup/1 Brick6: cfdstorage02-ib:/export/backup/1 Brick7: cfdstorage03-ib:/export/backup/1 Brick8: cfdstorage04-ib:/export/backup/1 Options Reconfigured: performance.io-thread-count: 16 auth.allow: 10.*,192.* Volume Name: main-volume Type: Distribute Status: Started Number of Bricks: 8 Transport-type: tcp,rdma Bricks: Brick1: cfdstorage01-ib:/export/main/0 Brick2: cfdstorage02-ib:/export/main/0 Brick3: cfdstorage03-ib:/export/main/0 Brick4: cfdstorage04-ib:/export/main/0 Brick5: cfdstorage01-ib:/export/main/1 Brick6: cfdstorage02-ib:/export/main/1 Brick7: cfdstorage03-ib:/export/main/1 Brick8: cfdstorage04-ib:/export/main/1 Options Reconfigured: performance.io-thread-count: 32 auth.allow: 10.*,192.* On 04/25/2012 07:10 AM, Bryan Whitehead wrote: > I'm confused, you said "everything works ok (IPoIB)" but later you > state you are using RDMA? Can you post details of your setup? Maybe > the output from gluster volume info<volumename>? > > On Sat, Apr 21, 2012 at 1:40 AM, Michael Mayer<michael at mayer.cx> wrote: >> Hi all, >> >> thanks for your suggestions, >> >> i think I have "solved" the performance issue now. I had a few too many >> kernel patches included. I am back to the stock RHEL 5.8 kernel with stock >> QLogic OFED and everything works ok (IPoIB). My original intent was to >> explore cachefs on RHEL5 by building a 2.6.32 kernel but while cachefs >> worked like a treat performance for gluster was as bad as reported >> previously - so will go without cachefs for now and reintroduce cachefs in >> an OS upgrade later on. >> >> I even have a nicely working rdma setup now and - using that - performance >> is 900 MB/s + and that consistently so. >> >> Since I have two volumes exported by the same bricks it seems I only can get >> one of them to use RDMA, the other will then refuse to mount and only mount >> if not using rdma on that one - but that is not a real problem for now as >> the second one is only used for backup purposes. >> >> Michael, >> >> On 04/12/2012 01:13 AM, Fabricio Cannini wrote: >> >> Hi there >> >> The only time i setup a gluster "distributed scratch" like Michael is doing, >> ( 3.0.5 Debian packages ) i too choose IPoIB simply because i could not get >> rdma working at all. >> Time was short and IPoIB "Just worked" well enough for our demand at the >> time, so i didn't looked into this issue. Plus, pinging and ssh'ing into a >> node through the IB interface comes handy when diagnosing and fixing >> networking issues. >> >> Em quarta-feira, 11 de abril de 2012, Sabuj Pattanayek<sabujp at gmail.com> >> escreveu: >>> I wonder if it's possible to have both rdma and ipoib served by a >>> single glusterfsd so I can test this? I guess so, since it's just a >>> tcp mount? >>> >>> On Wed, Apr 11, 2012 at 1:43 PM, Harry Mangalam<harry.mangalam at uci.edu> >>> wrote: >>>> On Tuesday 10 April 2012 15:47:08 Bryan Whitehead wrote: >>>> >>>>> with my infiniband setup I found my performance was much better by >>>>> setting up a TCP network over infiniband and then using pure tcp as >>>>> the transport with my gluster volume. For the life of me I couldn't >>>>> get rdma to beat tcp. >>>> Thanks for that data point, Brian. >>>> >>>> Very interesting. Is this a common experience? The RDMA experience has >>>> not >>>> been a very smooth one for me and doing everything with IPoIB would save >>>> a >>>> lot of headaches, especially if it's also higher performance. >>>> >>>> hjm >>>> >>>> -- >>>> >>>> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine >>>> >>>> [ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 >>>> >>>> 415 South Circle View Dr, Irvine, CA, 92697 [shipping] >>>> >>>> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) >>>> >>>> -- >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>
