I'm having some problems getting Glusterfs to run on my system. This is
the first time using it so you I'm not to knowledgeable yet. The system
I'm using has infiniband installed so I would like to use the IB for the
transport for better performance. Heres what I have done so far. I
first set up glusterfs to run over tcp on the Gig-E network and that
worked fine so I decided to move to the IB network and thats when my
problems started. Heres some information about the system.
Glusterfs: glusterfs-1.3.9
Fuse: fuse-2.7.3glfs10
Cards: Qlogic QLE7240
Drivers: InfiniPath-2.2
OS: Centos5.2
Kernel: 2.6.18-53
4 Storage nodes
46 Compute nodes
When I first ran configure on glusterfs I got the following message
after it ran configure.
GlusterFS configure summary
===========================
Fuse client : yes
Infiniband verbs : no
epoll IO multiplex : yes
If I look at lsmod I do have a ib_uverbs module installed and working
but the configure script doesn't pick it up. I decided move on and use
ib-sdp instead. I modified all my client and server volume files to
reflect ib-sdp for the transport then I started the server on my 4
storage nodes it started fine. I then went to one of my compute nodes
to mount the volume.
glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs
It returned with the following error.
glusterfs: could not open specfile
Then I tried to specify the volume file to see if that would work.
glusterfsd --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol
/glusterfs
It returned fine so I thought everything was working until I ran df I
got the following error.
df: `/glusterfs': Transport endpoint is not connected
I had it log the error that its getting when I try and mount the filesystem.
2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans: error:
not in progress - trace: Network is unreachable
2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer]
trans: not connected at the moment to submit frame type(2) op(4)
2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk] trans:
no proper reply from server, returning ENOTCONN
This where I'm now I looked around the devel list and didn't find
anything on using the qlogic drivers just the ofed IB drivers. Heres
what my configuration files. I would like to us ib-verbs if thats
possible the performance should be much better then ib-sdp.
volume posix-stripe
type storage/posix
option directory /export/glusterfs
end-volume
volume server
type protocol/server
option transport-type ib-sdp/server
option auth.ip.posix-stripe.allow 192.168.2.*
subvolumes posix-stripe
end-volume
volume client-stripe-1
type protocol/client
option transport-type ib-sdp/client
option remote-host storage-ib1.cluster
option remote-subvolume posix-stripe
end-volume
volume client-stripe-2
type protocol/client
option transport-type ib-sdp/client
option remote-host storage-ib2.cluster
option remote-subvolume posix-stripe
end-volume
volume client-stripe-3
type protocol/client
option transport-type ib-sdp/client
option remote-host storage-ib3.cluster
option remote-subvolume posix-stripe
end-volume
volume client-stripe-4
type protocol/client
option transport-type ib-sdp/client
option remote-host storage-ib4.cluster
option remote-subvolume posix-stripe
end-volume
volume stripe
type cluster/stripe
option block-size *:2MB # All files ending with .img are striped
with 2MB stripe block size.
subvolumes client-stripe-1 client-stripe-2 client-stripe-3
client-stripe-4
end-volume
-Peter
--
Peter Pelekis 5444 Napa Street
Western Scientific San Diego CA, 92110
Senior Systems Engineer www.wsm.com
pete.pelekis at wsm.com Fax 619-220-6590
Phone 619-220-6580 x212 Toll Free 800-443-6699
GSA# GS-35F-5009H
Visit us at:
"I'm a great believer in luck, and I find the harder I work the more I have of it."
-Thomas Jefferson
"If everything seems under control, you're not going fast enough."
-Mario Andretti
