Sorry the mail daemon just batched me the rest of this conversation and I see this is already done. please ignore. -Mic Mickey Mazarick wrote: > I had some difficulty getting OFED 1.3 working on kernel 2.6.27 about > 6 months back. It took some patching but I did find that you needed > to have the srq enabled for it to work. The ibv_srq_pingpong test app > was a good test for weather it would work with gluster of not. > > I also had to upgrade the firmware on the mellanox cards I have to > enable srq (send recieve que) > > -Mic > > Nathan Stratton wrote: >> >> Hate to post again, but anyone have any ideas on this? >> >> -Nathan >> >> On Fri, 18 Sep 2009, Nathan Stratton wrote: >> >>> >>> Has anyone been able to get Infiniband working with 2.6.31 kernel >>> and fuse 2.8.0? My config works fine on my Centos 2.6.18 box, so I >>> know that is ok. >>> >>> Infiniband looks good: >>> >>> [root at xen1 src]# lsmod |grep ib >>> ib_ucm 13752 0 >>> ib_uverbs 32256 2 rdma_ucm,ib_ucm >>> ib_ipoib 68880 0 >>> ib_mthca 123700 0 >>> >>> [root at xen1 src]# ibv_devices >>> device node GUID >>> ------ ---------------- >>> mthca0 0005ad00000327e8 >>> >>> Gluster looks like it starts OK, but I can't touch the mount and >>> after a while it times out. Debug logs: >>> >>> >>> [2009-09-18 19:36:17] D [glusterfsd.c:354:_get_specfp] glusterfs: >>> loading volume file /usr/local/etc/glusterfs/glusterfs.vol >>> ================================================================================ >>> >>> Version : glusterfs 2.0.6 built on Sep 18 2009 09:54:43 >>> TLA Revision : v2.0.6 >>> Starting Time: 2009-09-18 19:36:17 >>> Command line : glusterfs -L DEBUG -l /var/log/glusterfs.log >>> --disable-direct-io-mode /share >>> PID : 8303 >>> System name : Linux >>> Nodename : xen1.hou.blinkmind.com >>> Kernel Release : 2.6.31 >>> Hardware Identifier: x86_64 >>> >>> Given volfile: >>> +------------------------------------------------------------------------------+ >>> >>> 1: volume brick0 >>> 2: type protocol/client >>> 3: option transport-type ib-verbs/client >>> 4: option remote-host 172.16.0.200 >>> 5: option remote-port 6997 >>> 6: option transport.address-family inet/inet6 >>> 7: option remote-subvolume brick >>> 8: end-volume >>> 9: >>> 10: volume mirror0 >>> 11: type protocol/client >>> 12: option transport-type ib-verbs/client >>> 13: option remote-host 172.16.0.201 >>> 14: option remote-port 6997 >>> 15: option transport.address-family inet/inet6 >>> 16: option remote-subvolume brick >>> 17: end-volume >>> 18: >>> 19: volume brick1 >>> 20: type protocol/client >>> 21: option transport-type ib-verbs/client >>> 22: option remote-host 172.16.0.202 >>> 23: option remote-port 6997 >>> 24: option transport.address-family inet/inet6 >>> 25: option remote-subvolume brick >>> 26: end-volume >>> 27: >>> 28: volume mirror1 >>> 29: type protocol/client >>> 30: option transport-type ib-verbs/client >>> 31: option remote-host 172.16.0.203 >>> 32: option remote-port 6997 >>> 33: option transport.address-family inet/inet6 >>> 34: option remote-subvolume brick >>> 35: end-volume >>> 36: >>> 37: volume brick2 >>> 38: type protocol/client >>> 39: option transport-type ib-verbs/client >>> 40: option remote-host 172.16.0.204 >>> 41: option remote-port 6997 >>> 42: option transport.address-family inet/inet6 >>> 43: option remote-subvolume brick >>> 44: end-volume >>> 45: >>> 46: volume mirror2 >>> 47: type protocol/client >>> 48: option transport-type ib-verbs/client >>> 49: option remote-host 172.16.0.205 >>> 50: option remote-port 6997 >>> 51: option transport.address-family inet/inet6 >>> 52: option remote-subvolume brick >>> 53: end-volume >>> 54: >>> 55: volume block0 >>> 56: type cluster/replicate >>> 57: subvolumes brick0 mirror0 >>> 58: end-volume >>> 59: >>> 60: volume block1 >>> 61: type cluster/replicate >>> 62: subvolumes brick1 mirror1 >>> 63: end-volume >>> 64: >>> 65: volume block2 >>> 66: type cluster/replicate >>> 67: subvolumes brick2 mirror2 >>> 68: end-volume >>> 69: >>> 70: volume unify >>> 71: type cluster/distribute >>> 72: subvolumes block0 block1 block2 >>> 73: end-volume >>> 74: >>> >>> +------------------------------------------------------------------------------+ >>> >>> [2009-09-18 19:36:17] D [glusterfsd.c:1205:main] glusterfs: running >>> in pid 8303 >>> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick0: >>> defaulting frame-timeout to 30mins >>> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick0: >>> defaulting ping-timeout to 10 >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> brick0: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> brick0: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror0: >>> defaulting frame-timeout to 30mins >>> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror0: >>> defaulting ping-timeout to 10 >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> mirror0: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> mirror0: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick1: >>> defaulting frame-timeout to 30mins >>> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick1: >>> defaulting ping-timeout to 10 >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> brick1: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> brick1: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror1: >>> defaulting frame-timeout to 30mins >>> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror1: >>> defaulting ping-timeout to 10 >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> mirror1: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> mirror1: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick2: >>> defaulting frame-timeout to 30mins >>> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick2: >>> defaulting ping-timeout to 10 >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> brick2: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> brick2: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror2: >>> defaulting frame-timeout to 30mins >>> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror2: >>> defaulting ping-timeout to 10 >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> mirror2: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: >>> attempt to load file >>> /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so >>> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] >>> mirror2: no range check required for 'option remote-port 6997' >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got >>> GF_EVENT_PARENT_UP, attempting connect on transport >>> [2009-09-18 19:36:17] N [glusterfsd.c:1224:main] glusterfs: >>> Successfully started >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got >>> GF_EVENT_CHILD_UP >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] brick0: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] brick0: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] mirror0: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] mirror0: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] brick1: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] brick1: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] mirror1: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] mirror1: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] brick2: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] brick2: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] mirror2: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2: >>> bailing out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. >>> frame-timeout = 1800 >>> [2009-09-18 20:06:18] D >>> [client-protocol.c:5491:client_setvolume_cbk] mirror2: setvolume >>> failed (Transport endpoint is not connected) >>> [2009-09-18 20:06:18] D [dht-common.c:820:dht_lookup] unify: no >>> subvolume in layout for path=/, checking on all the subvols to see >>> if it is a directory >>> [2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: >>> lookup of / on block0 returned error (Transport endpoint is not >>> connected) >>> [2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: >>> lookup of / on block1 returned error (Transport endpoint is not >>> connected) >>> [2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: >>> lookup of / on block2 returned error (Transport endpoint is not >>> connected) >>> [2009-09-18 20:06:18] D [fuse-bridge.c:2385:fuse_root_lookup_cbk] >>> fuse: first lookup on root failed. >>> [2009-09-18 20:06:18] W [fuse-bridge.c:1841:fuse_statfs_cbk] >>> glusterfs-fuse: 2: ERR => -1 (Transport endpoint is not connected) >>> >>> >>> >>>> <> >>> Nathan Stratton CTO, BlinkMind, Inc. >>> nathan at robotics.net nathan at blinkmind.com >>> http://www.robotics.net http://www.blinkmind.com >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
