I'm going to answer my own question in case someone else wants to use
the latest open-ib code with gluster (has port bonding and some other
features I plan to try later).
SRQ is implemented in ofed 1.2 but needs the latest firmware from most
vendors to work. I was cursed with an off-brand ib card but it had a
mellanox chipset. After some tweaking we were able to patch the firmware
with a reference firmware from the mellanox site.
http://www.mellanox.com/support/firmware_table.php
After that everything went smoothly.
Thanks!
Mickey Mazarick wrote:
I didn't have that exact option, but I did have
ibv-recv-work-request-count set to 32. I also added the
ib-verbs-work-request-recv-count option as well. I've tried several
values there but I still get the same error. I'm running gluter as
root on a Centos 5 box upgraded to kernel 2.6.20
I actually found it here:
https://wiki.openfabrics.org/tiki-index.php?page=Downloading+Code+From+the+OFA+git+Repositories
git link: git://git.openfabrics.org/~vlad/ofed_1_2/.git
<git://git.openfabrics.org/%7Evlad/ofed_1_2/.git>
pre-patched with the latest infiniband code from openfabrics.
below is the volume spec I'm using as well.
Thanks for your help :-)
-Mic
### File: /etc/glusterfs-server.vol - GlusterFS Server Volume
Specification
volume main
type storage/posix
option directory /mnt/gluster/main
option debug off
end-volume
#volume main-io #iothreads can give performance a boost
# type performance/io-threads
# option thread-count 8
# subvolumes main
#end-volume
volume server
type protocol/server
# option transport-type tcp/server # For TCP/IP transport
# option transport-type ib-sdp/server # For Infiniband transport
option transport-type ib-verbs/server # For Infiniband Verbs transport
option ibv-recv-work-request-size 131072 # Higher performance if
its equal t o read-ahead size
option ibv-recv-work-request-count 32
option ibv-send-work-request-size 131072 # Higher performance if
its equal t o write-behind size
option ibv-send-work-request-count 32
option ib-verbs-work-request-recv-count 32
# subvolumes main-io
subvolumes main
option auth.ip.main.allow 192.168.*
end-volume
### END FILE ###
Anand Avati wrote:
Mickey,
what do you have as 'option ib-verbs-work-request-recv-count
<count>' in the server spec file? Also, are you trying to run
glusterfsd as non root? which is your distro?
thanks,
avati
2007/8/6, Mickey Mazarick <mic@xxxxxxxxxxxxxxxxxx
<mailto:mic@xxxxxxxxxxxxxxxxxx>>:
I'm having trouble getting the ibverbs transport working. I'm new
to the
infiniband world but the ib network is working correctly. I'm
using OFED
ver 1.2 but all the docs are for version 1.1.
The error in the gluster logs is:
[ib-verbs.c:1285:ib_verbs_get_device] transport/ib-verbs: server:
could
not create SRQ
Can someone point me in the right direction?
Thanks1
-Mickey Mazarick
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx <mailto:Gluster-devel@xxxxxxxxxx>
http://lists.nongnu.org/mailman/listinfo/gluster-devel
<http://lists.nongnu.org/mailman/listinfo/gluster-devel>
--
It always takes longer than you expect, even when you take into
account Hofstadter's Law.
-- Hofstadter's Law
--
