Hi Peter, Glusterfs ib-verbs transport needs libibverbs to be installed. Is it installed on your system. Also, in case of ib-sdp are you able to ping the other node using the interface which has ib-sdp support? regards, On Fri, Jul 25, 2008 at 2:26 AM, Peter Pelekis <pete.pelekis at wsm.com> wrote: > I'm having some problems getting Glusterfs to run on my system. This is > the first time using it so you I'm not to knowledgeable yet. The system > I'm using has infiniband installed so I would like to use the IB for the > transport for better performance. Heres what I have done so far. I > first set up glusterfs to run over tcp on the Gig-E network and that > worked fine so I decided to move to the IB network and thats when my > problems started. Heres some information about the system. > > Glusterfs: glusterfs-1.3.9 > Fuse: fuse-2.7.3glfs10 > Cards: Qlogic QLE7240 > Drivers: InfiniPath-2.2 > OS: Centos5.2 > Kernel: 2.6.18-53 > 4 Storage nodes > 46 Compute nodes > > When I first ran configure on glusterfs I got the following message > after it ran configure. > > GlusterFS configure summary > =========================== > Fuse client : yes > Infiniband verbs : no > epoll IO multiplex : yes > > If I look at lsmod I do have a ib_uverbs module installed and working > but the configure script doesn't pick it up. I decided move on and use > ib-sdp instead. I modified all my client and server volume files to > reflect ib-sdp for the transport then I started the server on my 4 > storage nodes it started fine. I then went to one of my compute nodes > to mount the volume. > glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs > > It returned with the following error. > glusterfs: could not open specfile > > Then I tried to specify the volume file to see if that would work. > glusterfsd --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol > /glusterfs > > It returned fine so I thought everything was working until I ran df I > got the following error. > df: `/glusterfs': Transport endpoint is not connected > > I had it log the error that its getting when I try and mount the > filesystem. > 2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans: error: > not in progress - trace: Network is unreachable > 2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer] > trans: not connected at the moment to submit frame type(2) op(4) > 2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk] trans: > no proper reply from server, returning ENOTCONN > > This where I'm now I looked around the devel list and didn't find > anything on using the qlogic drivers just the ofed IB drivers. Heres > what my configuration files. I would like to us ib-verbs if thats > possible the performance should be much better then ib-sdp. > > volume posix-stripe > type storage/posix > option directory /export/glusterfs > end-volume > > volume server > type protocol/server > option transport-type ib-sdp/server > option auth.ip.posix-stripe.allow 192.168.2.* > subvolumes posix-stripe > end-volume > > volume client-stripe-1 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib1.cluster > option remote-subvolume posix-stripe > end-volume > > volume client-stripe-2 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib2.cluster > option remote-subvolume posix-stripe > end-volume > > volume client-stripe-3 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib3.cluster > option remote-subvolume posix-stripe > end-volume > > volume client-stripe-4 > type protocol/client > option transport-type ib-sdp/client > option remote-host storage-ib4.cluster > option remote-subvolume posix-stripe > end-volume > > volume stripe > type cluster/stripe > option block-size *:2MB # All files ending with .img are striped > with 2MB stripe block size. > subvolumes client-stripe-1 client-stripe-2 client-stripe-3 > client-stripe-4 > end-volume > > -Peter > > -- > Peter Pelekis 5444 Napa Street > Western Scientific San Diego CA, 92110 > Senior Systems Engineer www.wsm.com > pete.pelekis at wsm.com Fax 619-220-6590 > Phone 619-220-6580 x212 Toll Free 800-443-6699 > GSA# GS-35F-5009H > > Visit us at: > > "I'm a great believer in luck, and I find the harder I work the more I have > of it." > -Thomas Jefferson > > "If everything seems under control, you're not going fast enough." > -Mario Andretti > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > -- Raghavendra G A centipede was happy quite, until a toad in fun, Said, "Prey, which leg comes after which?", This raised his doubts to such a pitch, He fell flat into the ditch, Not knowing how to run. -Anonymous -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zresearch.com/pipermail/gluster-users/attachments/20080725/02a8bba5/attachment.htm
