Hi,
I use the latest (master branch, upgrade at 2017/03/22) to build ceph with RDMA and use the fio to test its iops/latency/throughput.
In my environment, I setup 3 hosts and list the detail of each host below.
OS: ubuntu 16.04
Storage: SSD * 4 (256G * 4)
Memory: 64GB.
NICs: two NICs, one (intel 1G) for public network and the other (mellanox 10G) for private network.
There're 3 monitor and 24 osds equally distributed within 3 hosts which means each hosts contains 1 mon and 8 osds.
For my experiment, I use two configs, basic and RDMA.
Basic
[global] fsid = 0612cc7e-6239-456c-978b-b4df781fe831
mon initial members = ceph-1,ceph-2,ceph-3
mon host = 10.0.0.15,10.0.0.16,10.0.0.17
osd pool default size = 2
osd pool default pg num = 1024
osd pool default pgp num = 1024
RDMA
[global] fsid = 0612cc7e-6239-456c-978b-b4df781fe831
mon initial members = ceph-1,ceph-2,ceph-3
mon host = 10.0.0.15,10.0.0.16,10.0.0.17
osd pool default size = 2
osd pool default pg num = 1024
osd pool default pgp num = 1024
ms_type=async+rdma
ms_async_rdma_device_name = mlx4_0
What surprised me is that the result of RDMA mode is almost the same as the basic mode, the iops, latency, throughput, etc.
I also try to use different pattern of the fio parameter, such as read and write ratio, random operations or sequence operations.
All results are the same.
In order to figure out what's going on. I do the following steps.
1. Follow this article (https://community.mellanox.com/docs/DOC-2086) to make sure my RDMA environment.
2. To make sure the network traffic is transmitted by RDMA, I dump the traffic within the private network and the answear is yes. it use the RDMA.
3. Modify the ms_async_rdma_buffer_size to (256 << 10), no change.
4. Modfiy the ms_async_rdma_send_buffers to 2048, no change.
5. Modify the ms_async_rdma_receive_buffers to 2048, no change.
After above operations, I guess maybe my Ceph setup environment is not good for RDMA to improve the performance.
Do anyone know what kind of the ceph environment (replicated size, # of osd, # of mon, etc) is good for RDMA?
Thanks in advanced.
Best Regards,
Hung-Wei Chiu(邱宏瑋)
Hung-Wei Chiu(邱宏瑋)
--
Computer Center, Department of Computer Science
National Chiao Tung University
Computer Center, Department of Computer Science
National Chiao Tung University
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
