Hi Roman,
After only setting ms_cluster_type to "async+rdma", the ceph cluster
could be setup with below command:
OSD=3 MON=1 MDS=0 RGW=0 MGR=0 ../src/vstart.sh -n -d -X --msgr1
Have you also hit the problem when setting ms_public_type to "async+rdma"?
B.R.
Changcheng
On 14:45 Mon 15 Apr, Roman Penyaev wrote:
> On 2019-04-15 14:22, Liu, Changcheng wrote:
> > Hi Roman,
> > Thanks a lot for your time.
> >
> > I'm trying to use msg/async/rdma/iwarp with X722 network
> > interface card on ceph v14.2.0 version.
> >
> > Below is my previous configuration:
> > diff --git a/src/vstart.sh b/src/vstart.sh
> > index 22ca3c6318..bbae18c4ef 100755
> > --- a/src/vstart.sh
> > +++ b/src/vstart.sh
> > @@ -518,7 +518,21 @@ ms bind msgr1 = true
> > enable experimental unrecoverable data corrupting
> > features = *
> > osd_crush_chooseleaf_type = 0
> > debug asok assert abort = true
> > -$msgr_conf
> > +
> > + ms bind msgr2 = false
> > + ms bind msgr1 = true
> > +;tell the ceph use the AsyncMessenger + RDMA as your message
> > type
> > + ms_type = async+rdma
> > + ms_async_rdma_device_name = i40iw1
> > + ms_async_rdma_type = iwarp
> > + ms_async_rdma_support_srq = false
> > + ms_async_rdma_cm = true
> > +;; ms_async_rdma_port_num = 1
> > +; ms_async_rdma_send_buffers = 1024
> > +; ms_async_rdma_receive_buffers = 16384
> > +; ms_async_rdma_receive_queue_len = 1024
> > +;; ms_async_rdma_buffer_size = 4096
> > +
> > $extra_conf
> > EOF
> > if [ "$lockdep" -eq 1 ] ; then
> > (END)
> > I also tried to follow your suggestion to add below configuration
> > at the same time.
> > However, I still hit the same problem.
> > ms_type = async+rdma
> > ms_cluster_type = async+rdma
>
> No, what I meant is that ms_type should be async+posix, but ms_cluster_type
> is rdma, or just leave ms_cluster_type = async+rdma alone, i.e. just replace
> ms_type with ms_cluster_type:
>
> - ms_type = async+rdma
> + ms_cluster_type = async+rdma
>
> --
> Roman
>
