Thanks for the responses I got it to work with libibverbs. I found out you can install the OFED libibverbs with the infinipath drivers and it will work fine. Now I just need to tune the file system. What setting would you guys recommend for a system with 4 storage nodes using ib-verbs? The customer is going to be coping large data files back and forth between the storage and the compute nodes. -Peter Mickey Mazarick wrote: > All the stuff the you need for infiniband is in the latest OFED package: > http://www.openfabrics.org/downloads/OFED/ofed-1.3.1/OFED-1.3.1.tgz > > We are using Mellanox cards and also had to flash them up to the > newest version for ibverbs to work. > > -Mickey Mazarick > > > Peter Pelekis wrote: >> The Infinipath drivers don't have a libibverbs they call it >> libinfinipath. I'll test out the ib-sdp interface to see if it works. >> >> -Peter >> >> Raghavendra G wrote: >> >>> Hi Peter, >>> Glusterfs ib-verbs transport needs libibverbs to be installed. Is it >>> installed on your system. Also, in case of ib-sdp are you able to >>> ping the other node using the interface which has ib-sdp support? >>> >>> regards, >>> On Fri, Jul 25, 2008 at 2:26 AM, Peter Pelekis <pete.pelekis at wsm.com >>> <mailto:pete.pelekis at wsm.com>> wrote: >>> >>> I'm having some problems getting Glusterfs to run on my system. >>> This is >>> the first time using it so you I'm not to knowledgeable yet. The >>> system >>> I'm using has infiniband installed so I would like to use the IB >>> for the >>> transport for better performance. Heres what I have done so >>> far. I >>> first set up glusterfs to run over tcp on the Gig-E network and >>> that >>> worked fine so I decided to move to the IB network and thats >>> when my >>> problems started. Heres some information about the system. >>> >>> Glusterfs: glusterfs-1.3.9 >>> Fuse: fuse-2.7.3glfs10 >>> Cards: Qlogic QLE7240 >>> Drivers: InfiniPath-2.2 >>> OS: Centos5.2 >>> Kernel: 2.6.18-53 >>> 4 Storage nodes >>> 46 Compute nodes >>> >>> When I first ran configure on glusterfs I got the following message >>> after it ran configure. >>> >>> GlusterFS configure summary >>> =========================== >>> Fuse client : yes >>> Infiniband verbs : no >>> epoll IO multiplex : yes >>> >>> If I look at lsmod I do have a ib_uverbs module installed and >>> working >>> but the configure script doesn't pick it up. I decided move on >>> and use >>> ib-sdp instead. I modified all my client and server volume files to >>> reflect ib-sdp for the transport then I started the server on my 4 >>> storage nodes it started fine. I then went to one of my compute >>> nodes >>> to mount the volume. >>> glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs >>> >>> It returned with the following error. >>> glusterfs: could not open specfile >>> >>> Then I tried to specify the volume file to see if that would work. >>> glusterfsd >>> --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol >>> /glusterfs >>> >>> It returned fine so I thought everything was working until I ran >>> df I >>> got the following error. >>> df: `/glusterfs': Transport endpoint is not connected >>> >>> I had it log the error that its getting when I try and mount the >>> filesystem. >>> 2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans: >>> error: >>> not in progress - trace: Network is unreachable >>> 2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer] >>> trans: not connected at the moment to submit frame type(2) op(4) >>> 2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk] >>> trans: >>> no proper reply from server, returning ENOTCONN >>> >>> This where I'm now I looked around the devel list and didn't find >>> anything on using the qlogic drivers just the ofed IB drivers. >>> Heres >>> what my configuration files. I would like to us ib-verbs if thats >>> possible the performance should be much better then ib-sdp. >>> >>> volume posix-stripe >>> type storage/posix >>> option directory /export/glusterfs >>> end-volume >>> >>> volume server >>> type protocol/server >>> option transport-type ib-sdp/server >>> option auth.ip.posix-stripe.allow 192.168.2. >>> <http://192.168.2.>* >>> subvolumes posix-stripe >>> end-volume >>> >>> volume client-stripe-1 >>> type protocol/client >>> option transport-type ib-sdp/client >>> option remote-host storage-ib1.cluster >>> option remote-subvolume posix-stripe >>> end-volume >>> >>> volume client-stripe-2 >>> type protocol/client >>> option transport-type ib-sdp/client >>> option remote-host storage-ib2.cluster >>> option remote-subvolume posix-stripe >>> end-volume >>> >>> volume client-stripe-3 >>> type protocol/client >>> option transport-type ib-sdp/client >>> option remote-host storage-ib3.cluster >>> option remote-subvolume posix-stripe >>> end-volume >>> >>> volume client-stripe-4 >>> type protocol/client >>> option transport-type ib-sdp/client >>> option remote-host storage-ib4.cluster >>> option remote-subvolume posix-stripe >>> end-volume >>> >>> volume stripe >>> type cluster/stripe >>> option block-size *:2MB # All files ending with .img are >>> striped >>> with 2MB stripe block size. >>> subvolumes client-stripe-1 client-stripe-2 client-stripe-3 >>> client-stripe-4 >>> end-volume >>> >>> -Peter >>> >>> -- >>> Peter Pelekis 5444 Napa Street >>> Western Scientific San Diego CA, 92110 >>> Senior Systems Engineer www.wsm.com <http://www.wsm.com> >>> pete.pelekis at wsm.com <mailto:pete.pelekis at wsm.com> Fax >>> 619-220-6590 >>> Phone 619-220-6580 x212 Toll Free 800-443-6699 >>> GSA# GS-35F-5009H >>> >>> Visit us at: >>> >>> "I'm a great believer in luck, and I find the harder I work the >>> more I have of it." >>> -Thomas Jefferson >>> >>> "If everything seems under control, you're not going fast enough." >>> -Mario Andretti >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>> >>> >>> >>> >>> -- >>> Raghavendra G >>> >>> A centipede was happy quite, until a toad in fun, >>> Said, "Prey, which leg comes after which?", >>> This raised his doubts to such a pitch, >>> He fell flat into the ditch, >>> Not knowing how to run. >>> -Anonymous >>> >> >> >> > > -- Peter Pelekis 5444 Napa Street Western Scientific San Diego CA, 92110 Senior Systems Engineer www.wsm.com pete.pelekis at wsm.com Fax 619-220-6590 Phone 619-220-6580 x212 Toll Free 800-443-6699 GSA# GS-35F-5009H Visit us at: "I'm a great believer in luck, and I find the harder I work the more I have of it." -Thomas Jefferson "If everything seems under control, you're not going fast enough." -Mario Andretti
