Hi all,
I have multiple routers which connect to multiple upstream providers, I
have noticed a high latency shift in icmp (and generally all connection)
if I run b2 upload-file --threads 40 (and I can reproduce this)
What options do I have to analyze why this happens?
General Info:
Routers are connected between each other with 10G Mellanox Connect-X
cards via 10G SPF+ DAC cables via a 10G Switch from fs.com
Latency generally is around 0.18 ms between all routers (4).
Throughput is 9.4 Gbit/s with 0 retransmissions when tested with iperf3.
2 of the 4 routers are connected upstream with a 1G connection (separate
port, same network card)
All routers have the full internet routing tables, i.e. 80k entries for
IPv6 and 830k entries for IPv4
Conntrack is disabled (-j NOTRACK)
Kernel 5.4.60
2x Xeon X5670 @ 2.93 Ghz
96 GB RAM
No Swap
During high latency:
Latency on routers which have the traffic flow increases to 12 - 20 ms,
for all interfaces, moving of the stream (via bgp disable session) moves
also the high latency
iperf3 performance plumets to 300 - 400 MBits
CPU load (user / system) are around 0.1%
Ram Usage is around 3 - 4 GB
if_packets count is stable (around 8000 pkt/s more)
for b2 upload-file with 10 threads I can achieve 60 MB/s consistently,
with 40 threads the performance drops to 8 MB/s
I do not believe that 40 tcp streams should be any problem for a machine
of that size.
Thanks for any ideas, help, pointers, things I can verify / check /
provide additional!
Thomas
