On Thu, Mar 23, 2017 at 5:49 AM, Hung-Wei Chiu (邱宏瑋) <hwchiu@xxxxxxxxxxxxxx> wrote:
Hi,I use the latest (master branch, upgrade at 2017/03/22) to build ceph with RDMA and use the fio to test its iops/latency/throughput.In my environment, I setup 3 hosts and list the detail of each host below.OS: ubuntu 16.04Storage: SSD * 4 (256G * 4)Memory: 64GB.NICs: two NICs, one (intel 1G) for public network and the other (mellanox 10G) for private network.There're 3 monitor and 24 osds equally distributed within 3 hosts which means each hosts contains 1 mon and 8 osds.For my experiment, I use two configs, basic and RDMA.Basic[global] fsid = 0612cc7e-6239-456c-978b-b4df781fe831 mon initial members = ceph-1,ceph-2,ceph-3mon host = 10.0.0.15,10.0.0.16,10.0.0.17osd pool default size = 2osd pool default pg num = 1024osd pool default pgp num = 1024RDMA[global] fsid = 0612cc7e-6239-456c-978b-b4df781fe831 mon initial members = ceph-1,ceph-2,ceph-3mon host = 10.0.0.15,10.0.0.16,10.0.0.17osd pool default size = 2osd pool default pg num = 1024osd pool default pgp num = 1024ms_type=async+rdmams_async_rdma_device_name = mlx4_0What surprised me is that the result of RDMA mode is almost the same as the basic mode, the iops, latency, throughput, etc.I also try to use different pattern of the fio parameter, such as read and write ratio, random operations or sequence operations.All results are the same.
yes, most of latency comes from other components now.. although we still want to avoid extra copy in rdma side.
so current rdma backend only means it just can be choice compared to tcp/ip network. more benefits need to be get from others.
In order to figure out what's going on. I do the following steps.1. Follow this article (https://community.mellanox.com/docs/DOC-2086 ) to make sure my RDMA environment.2. To make sure the network traffic is transmitted by RDMA, I dump the traffic within the private network and the answear is yes. it use the RDMA.3. Modify the ms_async_rdma_buffer_size to (256 << 10), no change.4. Modfiy the ms_async_rdma_send_buffers to 2048, no change.5. Modify the ms_async_rdma_receive_buffers to 2048, no change.After above operations, I guess maybe my Ceph setup environment is not good for RDMA to improve the performance.Do anyone know what kind of the ceph environment (replicated size, # of osd, # of mon, etc) is good for RDMA?Thanks in advanced.Best Regards,
Hung-Wei Chiu(邱宏瑋)--
Computer Center, Department of Computer Science
National Chiao Tung University
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
