On 2019-04-16 10:58, Liu, Changcheng wrote:
Hi Roman,
After only setting ms_cluster_type to "async+rdma", the ceph cluster
could be setup with below command:
OSD=3 MON=1 MDS=0 RGW=0 MGR=0 ../src/vstart.sh -n -d -X --msgr1
Have you also hit the problem when setting ms_public_type to
"async+rdma"?
The problem happens exactly because you setup ms_public_type to rdma,
(RDMA connection to monitor is established *before* setuid() call)
ms_type compounds public and cluster, so setting ms_type to rdma you
setup cluster and public both to the rdma. Setting only cluster network
to rdma avoids the problem you described, because sockets descriptor
do not have that restriction of changing credentials, libe ueverbs
device has, so connection to monitor succeeds.
So in current master you are not able to use rdma for public network.
I can confirm, that this is broken.
But since all connections should be closed in get_monmap_and_config(),
for me is not clear why uverbs device is still opened and reused
afterwards. That should be debugged.
--
Roman
