Firstly, the OP's premise in asking, "Or should there be a differnce
of 10x", is fundamentally incorrect. Greater bandwidth does not mean
lower latency, though the latter almost always results in the former.
Unfortunately, changing the speed of light remains a difficult
engineering challenge :-). However, you can do things like: add
multiple links, overlap signals on the wire, and tweak error
correction encodings; all to get more bits on the wire without making
the wire itself any faster. Take Mellanox 100Gb ethernet, 1 lane is
25Gb, to get 50Gb they mash 2 lanes together, to get 100Gb they mash 4
lanes - the latency of single bit transmission is more-or-less
unchanged. Also note that with UDP/TCP pings or actual Ceph traffic
we're going via the kernel stack running on the CPU and as such the
speed & power-management of the CPU can make quite a difference.
Example 25GE on a dual-port CX-4 card in LACP bond, RHEL7 host.
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)
$ ofed_info | head -1
MLNX_OFED_LINUX-4.0-1.0.1.0 (OFED-4.0-1.0.1):
$ grep 'model name' /proc/cpuinfo | uniq
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
$ ibv_devinfo
hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 14.18.1000
node_guid: ...
sys_image_guid: ...
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
...
$ sudo ping -M do -s 8972 -c 100000 -f ...
100000 packets transmitted, 100000 received, 0% packet loss, time 4652ms
rtt min/avg/max/mdev = 0.029/0.031/2.711/0.015 ms, ipg/ewma 0.046/0.031 ms
$ sudo ping -M do -s 3972 -c 100000 -f ...
100000 packets transmitted, 100000 received, 0% packet loss, time 3321ms
rtt min/avg/max/mdev = 0.019/0.022/0.364/0.003 ms, ipg/ewma 0.033/0.022 ms
$ sudo ping -M do -s 1972 -c 100000 -f ...
100000 packets transmitted, 100000 received, 0% packet loss, time 2818ms
rtt min/avg/max/mdev = 0.017/0.018/0.086/0.005 ms, ipg/ewma 0.028/0.021 ms
$ sudo ping -M do -s 472 -c 100000 -f ...
100000 packets transmitted, 100000 received, 0% packet loss, time 2498ms
rtt min/avg/max/mdev = 0.014/0.016/0.305/0.005 ms, ipg/ewma 0.024/0.017 ms
$ sudo ping -M do -c 100000 -f ...
100000 packets transmitted, 100000 received, 0% packet loss, time 2363ms
rtt min/avg/max/mdev = 0.014/0.015/0.322/0.006 ms, ipg/ewma 0.023/0.016 ms
On 22 January 2018 at 22:37, Nick Fisk <nick@xxxxxxxxxx> wrote:
> Anyone with 25G ethernet willing to do the test? Would love to see what the
> latency figures are for that.
>
>
>
> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of
> Maged Mokhtar
> Sent: 22 January 2018 11:28
> To: ceph-users@xxxxxxxxxxxxxx
> Subject: Re: What is the should be the expected latency of
> 10Gbit network connections
>
>
>
> On 2018-01-22 08:39, Wido den Hollander wrote:
>
>
>
> On 01/20/2018 02:02 PM, Marc Roos wrote:
>
> If I test my connections with sockperf via a 1Gbit switch I get around
> 25usec, when I test the 10Gbit connection via the switch I have around
> 12usec is that normal? Or should there be a differnce of 10x.
>
>
> No, that's normal.
>
> Tests with 8k ping packets over different links I did:
>
> 1GbE: 0.800ms
> 10GbE: 0.200ms
> 40GbE: 0.150ms
>
> Wido
>
>
> sockperf ping-pong
>
> sockperf: Warmup stage (sending a few dummy messages)...
> sockperf: Starting test...
> sockperf: Test end (interrupted by timer)
> sockperf: Test ended
> sockperf: [Total Run] RunTime=10.100 sec; SentMessages=432875;
> ReceivedMessages=432874
> sockperf: ========= Printing statistics for Server No: 0
> sockperf: [Valid Duration] RunTime=10.000 sec; SentMessages=428640;
> ReceivedMessages=428640
> sockperf: ====> avg-lat= 11.609 (std-dev=1.684)
> sockperf: # dropped messages = 0; # duplicated messages = 0; #
> out-of-order messages = 0
> sockperf: Summary: Latency is 11.609 usec
> sockperf: Total 428640 observations; each percentile contains 4286.40
> observations
> sockperf: ---> <MAX> observation = 856.944
> sockperf: ---> percentile 99.99 = 39.789
> sockperf: ---> percentile 99.90 = 20.550
> sockperf: ---> percentile 99.50 = 17.094
> sockperf: ---> percentile 99.00 = 15.578
> sockperf: ---> percentile 95.00 = 12.838
> sockperf: ---> percentile 90.00 = 12.299
> sockperf: ---> percentile 75.00 = 11.844
> sockperf: ---> percentile 50.00 = 11.409
> sockperf: ---> percentile 25.00 = 11.124
> sockperf: ---> <MIN> observation = 8.888
>
> sockperf: Warmup stage (sending a few dummy messages)...
> sockperf: Starting test...
> sockperf: Test end (interrupted by timer)
> sockperf: Test ended
> sockperf: [Total Run] RunTime=1.100 sec; SentMessages=22065;
> ReceivedMessages=22064
> sockperf: ========= Printing statistics for Server No: 0
> sockperf: [Valid Duration] RunTime=1.000 sec; SentMessages=20056;
> ReceivedMessages=20056
> sockperf: ====> avg-lat= 24.861 (std-dev=1.774)
> sockperf: # dropped messages = 0; # duplicated messages = 0; #
> out-of-order messages = 0
> sockperf: Summary: Latency is 24.861 usec
> sockperf: Total 20056 observations; each percentile contains 200.56
> observations
> sockperf: ---> <MAX> observation = 77.158
> sockperf: ---> percentile 99.99 = 54.285
> sockperf: ---> percentile 99.90 = 37.864
> sockperf: ---> percentile 99.50 = 34.406
> sockperf: ---> percentile 99.00 = 33.337
> sockperf: ---> percentile 95.00 = 27.497
> sockperf: ---> percentile 90.00 = 26.072
> sockperf: ---> percentile 75.00 = 24.618
> sockperf: ---> percentile 50.00 = 24.443
> sockperf: ---> percentile 25.00 = 24.361
> sockperf: ---> <MIN> observation = 16.746
> [root@c01 sbin]# sockperf ping-pong -i 192.168.0.12 -p 5001 -t 10
> sockperf: == version #2.6 ==
> sockperf[CLIENT] send on:sockperf: using recvfrom() to block on
> socket(s)
>
>
>
>
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> I find the ping command with flood option handy to measure latency, gives
> stats min/max/average/std deviation
>
> example:
>
> ping -c 100000 -f 10.0.1.12
>
> Maged
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Cheers,
~Blairo
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
