Hello All,
I'm having trouble mounting gluster volumes over RDMA. Everything seems to work fine until I try to mount the gluster volume. At this point it seems to crash glusterd and give me an error "could not create QP [Permission Denied]. If I change change the transport to tcp, everything works as expected. I am pretty new to Infiniband and RDMA, so this may very well be something I'm doing wrong. That said, I was able to get NFSoRDMA (and qperf) working on these same hosts, so I'm confident my hardware is working. This is on CentOS 7.5 using gluster 3.12.15 from the CentOS repository. I can't find anything regarding this specific error and any help is much appreciated.
# rdma works...
[root@gluster1 ~]# qperf --use_bits_per_sec gluster2-ib rc_bi_bw
rc_bi_bw:
bw = 35.3 Gb/sec
# NFSoRDMA works....
[root@gluster1 data]# mount -t nfs gluster2-ib:/test /mnt/test -o rdma,port=20049
[root@gluster1 ~]#
[root@gluster1 data]# df -h|grep root
/dev/mapper/centos_gluster1-root 33G 7.8G 26G 24% /
gluster2-ib:/test 33G 29G 4.1G 88% /mnt/test
[root@gluster1 ~]# gluster volume create data replica 3 transport rdma gluster1-ib:/zp1/data vmh2-ib:/zp1/data vmh3-ib:/zp1/data force
volume create: data: success: please start the volume to access data
[root@gluster1 ~]#
[root@gluster1 ~]# gluster volume start data
volume start: data: success
[root@gluster1 ~]#
[root@gluster1 ~]# gluster volume info data
Volume Name: data
Type: Replicate
Volume ID: 6f32999c-2d20-4fc2-838b-82972171df7e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: rdma
Bricks:
Brick1: gluster1-ib:/zp1/data
Brick2: gluster2-ib:/zp1/data
Brick3: gluster3-ib:/zp1/data
Options Reconfigured:
nfs.disable: on
performance.client-io-threads: off
[root@gluster1 ~]#
[root@gluster1 ~]# mount -t glusterfs -o transport=rdma gluster1:/data /mnt/data
Mount failed. Please check the log file for more details.
[root@gluster1 ~]#
[root@gluster1 ~]# gluster volume info data
Connection failed. Please check if gluster daemon is operational.
[root@gluster1 ~]#
[root@gluster1 ~]# tail -n 30 /var/log/messages|grep glusterd
Nov 10 11:18:07 gluster1 glusterd[1881]: [2018-11-10 16:18:07.457818] C [MSGID: 103021] [rdma.c:3263:gf_rdma_create_qp] 0-rdma.management: rdma.management: could not create QP [Permission denied]
Nov 10 11:18:07 gluster1 glusterd[1881]: pending frames:
Nov 10 11:18:07 gluster1 glusterd[1881]: patchset: git://git.gluster.org/glusterfs.git
Nov 10 11:18:07 gluster1 glusterd[1881]: signal received: 11
Nov 10 11:18:07 gluster1 glusterd[1881]: time of crash:
Nov 10 11:18:07 gluster1 glusterd[1881]: 2018-11-10 16:18:07
Nov 10 11:18:07 gluster1 glusterd[1881]: configuration details:
Nov 10 11:18:07 gluster1 glusterd[1881]: argp 1
Nov 10 11:18:07 gluster1 glusterd[1881]: backtrace 1
Nov 10 11:18:07 gluster1 glusterd[1881]: dlfcn 1
Nov 10 11:18:07 gluster1 glusterd[1881]: libpthread 1
Nov 10 11:18:07 gluster1 glusterd[1881]: llistxattr 1
Nov 10 11:18:07 gluster1 glusterd[1881]: setfsid 1
Nov 10 11:18:07 gluster1 glusterd[1881]: spinlock 1
Nov 10 11:18:07 gluster1 glusterd[1881]: epoll.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: xattr.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: st_atim.tv_nsec 1
Nov 10 11:18:07 gluster1 glusterd[1881]: package-string: glusterfs 3.12.15
Nov 10 11:18:07 gluster1 glusterd[1881]: ---------
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
