Re: transport/ib-verbs: server: could not create SRQ

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I'm going to answer my own question in case someone else wants to use the latest open-ib code with gluster (has port bonding and some other features I plan to try later).

SRQ is implemented in ofed 1.2 but needs the latest firmware from most vendors to work. I was cursed with an off-brand ib card but it had a mellanox chipset. After some tweaking we were able to patch the firmware with a reference firmware from the mellanox site.
http://www.mellanox.com/support/firmware_table.php

After that everything went smoothly.

Thanks!


Mickey Mazarick wrote:
I didn't have that exact option, but I did have ibv-recv-work-request-count set to 32. I also added the ib-verbs-work-request-recv-count option as well. I've tried several values there but I still get the same error. I'm running gluter as root on a Centos 5 box upgraded to kernel 2.6.20 I actually found it here: https://wiki.openfabrics.org/tiki-index.php?page=Downloading+Code+From+the+OFA+git+Repositories git link: git://git.openfabrics.org/~vlad/ofed_1_2/.git <git://git.openfabrics.org/%7Evlad/ofed_1_2/.git>

pre-patched with the latest infiniband code from openfabrics.
below is the volume spec I'm using as well.

Thanks for your help :-)

-Mic

### File: /etc/glusterfs-server.vol - GlusterFS Server Volume Specification

volume main
 type storage/posix
 option directory /mnt/gluster/main
 option debug off
end-volume

#volume main-io    #iothreads can give performance a boost
#   type performance/io-threads
#   option thread-count 8
#   subvolumes main
#end-volume

volume server
 type protocol/server
#  option transport-type tcp/server     # For TCP/IP transport
# option transport-type ib-sdp/server  # For Infiniband transport
option transport-type ib-verbs/server # For Infiniband Verbs transport
option ibv-recv-work-request-size 131072 # Higher performance if its equal t o read-ahead size
option ibv-recv-work-request-count  32
option ibv-send-work-request-size 131072 # Higher performance if its equal t o write-behind size
option ibv-send-work-request-count  32
option ib-verbs-work-request-recv-count 32

#  subvolumes main-io
subvolumes main
 option auth.ip.main.allow 192.168.*
end-volume

### END FILE ###



Anand Avati wrote:
Mickey,
what do you have as 'option ib-verbs-work-request-recv-count <count>' in the server spec file? Also, are you trying to run glusterfsd as non root? which is your distro?

thanks,
avati

2007/8/6, Mickey Mazarick <mic@xxxxxxxxxxxxxxxxxx <mailto:mic@xxxxxxxxxxxxxxxxxx>>:

    I'm having trouble getting the ibverbs transport working. I'm new
    to the
    infiniband world but the ib network is working correctly. I'm
    using OFED
    ver 1.2 but all the docs are for version 1.1.

    The error in the gluster logs is:
    [ib-verbs.c:1285:ib_verbs_get_device] transport/ib-verbs: server:
    could
    not create SRQ

    Can someone point me in the right direction?

    Thanks1
    -Mickey Mazarick



    _______________________________________________
    Gluster-devel mailing list
    Gluster-devel@xxxxxxxxxx <mailto:Gluster-devel@xxxxxxxxxx>
    http://lists.nongnu.org/mailman/listinfo/gluster-devel
    <http://lists.nongnu.org/mailman/listinfo/gluster-devel>




--
It always takes longer than you expect, even when you take into account Hofstadter's Law.

-- Hofstadter's Law




--



[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux