The Infinipath drivers don't have a libibverbs they call it libinfinipath. I'll test out the ib-sdp interface to see if it works. -Peter Raghavendra G wrote: > Hi Peter, > Glusterfs ib-verbs transport needs libibverbs to be installed. Is it > installed on your system. Also, in case of ib-sdp are you able to ping > the other node using the interface which has ib-sdp support? > > regards, > On Fri, Jul 25, 2008 at 2:26 AM, Peter Pelekis <pete.pelekis at wsm.com > <mailto:pete.pelekis at wsm.com>> wrote: > > I'm having some problems getting Glusterfs to run on my system. > This is > the first time using it so you I'm not to knowledgeable yet. The > system > I'm using has infiniband installed so I would like to use the IB > for the > transport for better performance. Heres what I have done so far. I > first set up glusterfs to run over tcp on the Gig-E network and that > worked fine so I decided to move to the IB network and thats when my > problems started. Heres some information about the system. > > Glusterfs: glusterfs-1.3.9 > Fuse: fuse-2.7.3glfs10 > Cards: Qlogic QLE7240 > Drivers: InfiniPath-2.2 > OS: Centos5.2 > Kernel: 2.6.18-53 > 4 Storage nodes > 46 Compute nodes > > When I first ran configure on glusterfs I got the following message > after it ran configure. > > GlusterFS configure summary > =========================== > Fuse client : yes > Infiniband verbs : no > epoll IO multiplex : yes > > If I look at lsmod I do have a ib_uverbs module installed and working > but the configure script doesn't pick it up. I decided move on and use > ib-sdp instead. I modified all my client and server volume files to > reflect ib-sdp for the transport then I started the server on my 4 > storage nodes it started fine. I then went to one of my compute nodes > to mount the volume. > glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs > > It returned with the following error. > glusterfs: could not open specfile > > Then I tried to specify the volume file to see if that would work. > glusterfsd --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol > /glusterfs > > It returned fine so I thought everything was working until I ran df I > got the following error. > df: `/glusterfs': Transport endpoint is not connected > > I had it log the error that its getting when I try and mount the > filesystem. > 2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans: > error: > not in progress - trace: Network is unreachable > 2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer] > trans: not connected at the moment to submit frame type(2) op(4) > 2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk] > trans: > no proper reply from server, returning ENOTCONN > > This where I'm now I looked around the devel list and didn't find > anything on using the qlogic drivers just the ofed IB drivers. Heres > what my configuration files. I would like to us ib-verbs if thats > possible the performance should be much better then ib-sdp. > > volume posix-stripe > type storage/posix > option directory /export/glusterfs > end-volume > > volume server > type protocol/server > option transport-type ib-sdp/server > option auth.ip.posix-stripe.allow 192.168.2. > <http://192.168.2.>* > subvolumes posix-stripe > end-volume > > volume client-stripe-1 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib1.cluster > option remote-subvolume posix-stripe > end-volume > > volume client-stripe-2 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib2.cluster > option remote-subvolume posix-stripe > end-volume > > volume client-stripe-3 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib3.cluster > option remote-subvolume posix-stripe > end-volume > > volume client-stripe-4 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib4.cluster > option remote-subvolume posix-stripe > end-volume > > volume stripe > type cluster/stripe > option block-size *:2MB # All files ending with .img are striped > with 2MB stripe block size. > subvolumes client-stripe-1 client-stripe-2 client-stripe-3 > client-stripe-4 > end-volume > > -Peter > > -- > Peter Pelekis 5444 Napa Street > Western Scientific San Diego CA, 92110 > Senior Systems Engineer www.wsm.com <http://www.wsm.com> > pete.pelekis at wsm.com <mailto:pete.pelekis at wsm.com> Fax > 619-220-6590 > Phone 619-220-6580 x212 Toll Free 800-443-6699 > GSA# GS-35F-5009H > > Visit us at: > > "I'm a great believer in luck, and I find the harder I work the > more I have of it." > -Thomas Jefferson > > "If everything seems under control, you're not going fast enough." > -Mario Andretti > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > > -- > Raghavendra G > > A centipede was happy quite, until a toad in fun, > Said, "Prey, which leg comes after which?", > This raised his doubts to such a pitch, > He fell flat into the ditch, > Not knowing how to run. > -Anonymous -- Peter Pelekis 5444 Napa Street Western Scientific San Diego CA, 92110 Senior Systems Engineer www.wsm.com pete.pelekis at wsm.com Fax 619-220-6590 Phone 619-220-6580 x212 Toll Free 800-443-6699 GSA# GS-35F-5009H Visit us at: "I'm a great believer in luck, and I find the harder I work the more I have of it." -Thomas Jefferson "If everything seems under control, you're not going fast enough." -Mario Andretti
