Re: The performance of ceph with RDMA

Haomai Wang <haomai@xxxxxxxx> · Fri, 24 Mar 2017 06:43:34 -0400

OH.. you can refer to performance related threads on ceph/ceph-devel maillist to get ssd-optimized ceph.conf. the default conf lack of good support on ssd.

On Fri, Mar 24, 2017 at 6:33 AM, Hung-Wei Chiu (邱宏瑋) <hwchiu@xxxxxxxxxxxxxx> wrote:
Hi.

Basic
[global]                                                                                                                                             
fsid = 0612cc7e-6239-456c-978b-b4df781fe831
mon initial members = ceph-1,ceph-2,ceph-3
mon host = 10.0.0.15,10.0.0.16,10.0.0.17
osd pool default size = 2
osd pool default pg num = 1024
osd pool default pgp num = 1024

RDMA
[global]                                                                                                                                             
fsid = 0612cc7e-6239-456c-978b-b4df781fe831
mon initial members = ceph-1,ceph-2,ceph-3
mon host = 10.0.0.15,10.0.0.16,10.0.0.17
osd pool default size = 2
osd pool default pg num = 1024
osd pool default pgp num = 1024
ms_type=async+rdma
ms_async_rdma_device_name = mlx4_0

Thanks.

Best Regards,

Hung-Wei Chiu(邱宏瑋)--
Computer Center, Department of Computer Science
National Chiao Tung University

2017-03-24 18:28 GMT+08:00 Haomai Wang <haomai@xxxxxxxx>:
the content of  ceph.conf ?

On Fri, Mar 24, 2017 at 4:32 AM, Hung-Wei Chiu (邱宏瑋) <hwchiu@xxxxxxxxxxxxxx> wrote:
Hi Deepak.

Thansk your reply,

I try to use gperf to profile the ceph-osd with basic mode (without RDMA) and you can see the result in the following link.
http://imgur.com/a/SJgEL

In the gperf result, we can see the whole CPU usage can divided into three significant part, (Network: 36%, FileStore: 17% PG related:29%).
Do you have any idea about this?

Thanks.

Best Regards,

Hung-Wei Chiu(邱宏瑋)--
Computer Center, Department of Computer Science
National Chiao Tung University

2017-03-24 14:31 GMT+08:00 Deepak Naidu <dnaidu@xxxxxxxxxx>:
>>Now, the kernel can decrease the CPU cycle usages for network I/O processing because of RDMA enabled (right?).

Yes, kernel should have comparatively free cycles when using RDMA over TCP

>>does it means host can provide more CPU for other processing, such as Disk I/O ?

This can be subjective, bcos let's say u have an IO process which takes 10 mins over TCP then when using RDMA it might be 3 mins as your CPU cycles are less used by RDMA but if ur application process which provides the RDMA functionality might have processing cycles as well, which might use CPU cycles as well, example client or fuse plugins on the host getting it ?

--

Deepak

> On Mar 23, 2017, at 11:10 PM, Hung-Wei Chiu (邱宏瑋) <hwchiu@xxxxxxxxxxxxxx> wrote:

>

> Now, the kernel can decrease the CPU cycle usages for network I/O processing because of RDMA enabled (right?).

> does it means host can provide more CPU for other processing, such as Disk I/O ?

-----------------------------------------------------------------------------------

This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution

is prohibited.  If you are not the intended recipient, please contact the sender by

reply email and destroy all copies of the original message.

-----------------------------------------------------------------------------------

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com