Whenever I try to start or mount a GlusterFS 3.1.1 volume that uses RDMA, I'm seeing the following error messages in the log file on the server: [2010-11-30 18:37:53.51270] I [nfs.c:652:init] nfs: NFS service started [2010-11-30 18:37:53.51362] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-11-30 18:37:53.51375] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-11-30 18:37:53.59628] E [rdma.c:2066:rdma_create_cq] rpc-transport/rdma: testdir-client-0: creation of send_cq failed [2010-11-30 18:37:53.59851] E [rdma.c:3771:rdma_get_device] rpc-transport/rdma: testdir-client-0: could not create CQ [2010-11-30 18:37:53.59925] E [rdma.c:3957:rdma_init] rpc-transport/rdma: could not create rdma device for mthca0 [2010-11-30 18:37:53.60009] E [rdma.c:4789:init] testdir-client-0: Failed to initialize IB Device [2010-11-30 18:37:53.60030] E [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma' initialization failed On the client, I see: [2010-11-30 18:43:49.653469] W [io-stats.c:1644:init] testdir: dangling volume. check volfile [2010-11-30 18:43:49.653573] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-11-30 18:43:49.653607] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-11-30 18:43:49.736275] E [rdma.c:2066:rdma_create_cq] rpc-transport/rdma: testdir-client-0: creation of send_cq failed [2010-11-30 18:43:49.736651] E [rdma.c:3771:rdma_get_device] rpc-transport/rdma: testdir-client-0: could not create CQ [2010-11-30 18:43:49.736689] E [rdma.c:3957:rdma_init] rpc-transport/rdma: could not create rdma device for mthca0 [2010-11-30 18:43:49.736805] E [rdma.c:4789:init] testdir-client-0: Failed to initialize IB Device [2010-11-30 18:43:49.736841] E [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma' initialization failed This results in an unsuccessful mount. I created the mount using the following commands: /usr/local/glusterfs/3.1.1/sbin/gluster volume create testdir transport rdma submit-1:/exports /usr/local/glusterfs/3.1.1/sbin/gluster volume start testdir To mount the directory, I use: mount -t glusterfs submit-1:/testdir /mnt/glusterfs I don't think it is an Infiniband problem since GlusterFS 3.0.6 and GlusterFS 3.1.0 worked on the same systems. For GlusterFS 3.1.0, the commands listed above produced no error messages. If anyone can provide help with debugging these error messages, it would be appreciated.
