Re: The performance of ceph with RDMA

[Deepak Naidu <dnaidu@xxxxxxxxxx>]

RDMA is of interest to me. So my below comment.

 

>> What surprised me is that the result of RDMA mode is almost the same as the basic mode, the iops, latency, throughput, etc.

 

Pardon  my knowledge here. If I read your ceph.conf and your notes. It seems that you are using RDMA only for “cluster/private network” ? so how do you expect RDMA to be efficient on client IOPS/latency/throughput.

 

 

--

Deepak

 

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Haomai Wang
Sent: Thursday, March 23, 2017 4:34 AM
To: Hung-Wei Chiu (邱宏瑋)
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] The performance of ceph with RDMA

 

 

 

On Thu, Mar 23, 2017 at 5:49 AM, Hung-Wei Chiu (邱宏瑋) <hwchiu@xxxxxxxxxxxxxx> wrote:

Hi,

 

I use the latest (master branch, upgrade at 2017/03/22) to build ceph with RDMA and use the fio to test its iops/latency/throughput.

 

In my environment, I setup 3 hosts and list the detail of each host below.

 

OS: ubuntu 16.04

Storage: SSD * 4 (256G * 4)

Memory: 64GB.

NICs: two NICs, one (intel 1G) for public network and the other (mellanox 10G) for private network.

 

There're 3 monitor and 24 osds equally distributed within 3 hosts which means each hosts contains 1 mon and 8 osds.

 

For my experiment, I use two configs, basic and RDMA.

 

Basic

[global]                                                                                                                                             fsid = 0612cc7e-6239-456c-978b-b4df781fe831

mon initial members = ceph-1,ceph-2,ceph-3

mon host = 10.0.0.15,10.0.0.16,10.0.0.17

osd pool default size = 2

osd pool default pg num = 1024

osd pool default pgp num = 1024

 

 

RDMA

[global]                                                                                                                                             fsid = 0612cc7e-6239-456c-978b-b4df781fe831

mon initial members = ceph-1,ceph-2,ceph-3

mon host = 10.0.0.15,10.0.0.16,10.0.0.17

osd pool default size = 2

osd pool default pg num = 1024

osd pool default pgp num = 1024

ms_type=async+rdma

ms_async_rdma_device_name = mlx4_0

 

 

What surprised me is that the result of RDMA mode is almost the same as the basic mode, the iops, latency, throughput, etc.

I also try to use different pattern of the fio parameter, such as read and write ratio, random operations or sequence operations. 

All results are the same.

 

yes, most of latency comes from other components now.. although we still want to avoid extra copy in rdma side. 

 

so current rdma backend only means it just can be choice compared to tcp/ip network. more benefits need to be get from others.

 

 

In order to figure out what's going on. I do the following steps.

 

1. Follow this article (https://community.mellanox.com/docs/DOC-2086) to make sure my RDMA environment.

2. To make sure the network traffic is transmitted by RDMA, I dump the traffic within the private network and the answear is yes. it use the RDMA.

3. Modify the ms_async_rdma_buffer_size to (256 << 10), no change.

4. Modfiy the ms_async_rdma_send_buffers to 2048, no change.

5. Modify the ms_async_rdma_receive_buffers to 2048, no change.

 

After above operations, I guess maybe my Ceph setup environment is not good for RDMA to improve the performance.

 

Do anyone know what kind of the ceph environment (replicated size, # of osd, # of mon, etc) is good for RDMA?

 

Thanks in advanced.

 

 

Best Regards,

Hung-Wei Chiu(邱宏瑋)

--
Computer Center, Department of Computer Science
National Chiao Tung University

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

---

This email message is for the sole use of the intended recipient(s) and may contain confidential information.  Any unauthorized review, use, disclosure or distribution is prohibited.  If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

---

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com