Hi,
I am experimenting with LFN using netem. My setup consists of two
linux (sles10/2.6.16/64-bit) boxes connected via GigE link (no switch
in between) to another linux box (sles10/64-bit) that acts has a
router. I add delay (and other link characterstics) via netem on the
router. Something like this:
[ sender ] ---- gige ---- [ router ] ----- gige ---- [ receiver ]
192.1.1.2 1.1.1 1.2.1 192.1.2.2
eth1 eth2
sender and receiver have Broadcom/NetX II/Gige NICs. and the router
has e1000 NICs. and TSO is turned-on on all NICs.
I insert equal delay using netem on the router to both eth1 and eth2
interfaces. So to test with 100ms RTT, I add 50ms delay to outgoing
traffic on both eth1 and eth2.
If I remove "netem" altogether, the RTT is around 0.1 ms (LAN times)
between sender and receiver and I get a healthy ~110MBps thruput.
tcpdump trace: http://slashdev.googlepages.com/trace.nuttcp.0ms.gz
If I add 20ms RTT, the tcp thruput is still around 95Mbps (good).
If I add 100ms RTT, the tcp thruput crawls to 230KBps :-(
tcpdump trace: http://slashdev.googlepages.com/trace.nuttcp.100ms.gz
Note: the trace was collected from "eth2" on the router.
I've tuned the tcp buffers (tcp_r|wmem) to have 15MB as max value (to
account for large BDP) on both sender and receiver. And turned on
usual performance tunables (timestamps, sacks etc.). My app (nuttcp)
does not set socket buffer sizes explicitly (verified via strace(8)),
so auto-tuning is active. And I've set congestion control algorithm to
"cubic" (tried "bic" and "reno" with similar results).
Given all this, I am not able to deduce the cause of poor TCP
performance with 100ms RTT. tcptrace (tcptrace.org) for the 100ms dump
shows lot of ooo packets. Is that the reason? Or is it something else?
Any hints and advice on fixing my setup? Or anything I need to
check/fix on the router? I'd like to see linux tcp fill the whole
100ms/1Gbps link :-)
If you need any more details please let me.
Thanks in advance,
S.
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
