Hi all,
I'm working on using rdma to improve message transaction performance
on distributed storage system(Ceph) development.
Does anyone know what's the right tool to compare RDMA vs TCP peformance?
Such as bandwidth, latency. Especially the tool that could measure
the time to transact the same data size.
Previously, I use iperf & ib_send_bw to do test(send same data size).
However, it shows that ib_send_bw use more time to send data than
iperf.
nstcc1@nstcloudcc1:~$ time ib_send_bw -c RC -d rocep4s0 -i 1 -p 18515 -q 1 -r 4096 -t 1024 -s 1073741824 --report_gbits -F 192.168.199.222
real 3m53.858s
user 3m48.456s
sys 0m5.318s
nstcc1@nstcloudcc1:~$ time iperf -c 192.168.199.222 -p 8976 -n 1073741824 -P 1
real 0m1.688s
user 0m0.020s
sys 0m1.644s
In Ceph, the result shows that rdma performance (RC transaction type,
SEDN operation) is worse or not much better than TCP implemented performance.
Test A:
1 client thread send 20GB data to 1 server thread (marked as 1C:1S)
Result:
1) implementation based on RDMA
Take 171.921294s to finish send 20GB data.
2) implementation based on TCP
Take 62.444163s to finish send 20GB data.
Test B:
16 client threads send 16x20GB data to 1 server thread (marked as 16C:1S)
Result:
1) implementation base on RDMA
Take 261.285612s to finish send 16x20GB data.
2) implementation based on TCP
Take 318.949126 to finish send 16x20GB data.
B.R.
Changcheng
