Hi, Thanks! On Thu, 2019-12-05 at 17:05 -0800, Kan Yan wrote: > > Anyhow, do you have any tips on debugging this? This is still without > > AQL code. The AQM stats for the AP look fine, basically everything is 0 > > except for "new-flows", "tx-bytes" and "tx-packets". > > If the "backlog" field is also 0, then it is a sign that the TCP stack > is not feeding packets fast enough. Mostly, not always. One thing I do noticed there now is that I get like 64k packets into the driver. Maybe somehow TSO is interacting badly with the TXQs and the tracking here, since TSO makes the traffic *very* bursty? A 64k packet in the driver will typically expand to 9 or 10 A-MSDUs I think? > > One thing that does seem odd is that the new-flows counter is increasing > > this rapidly - shouldn't we expect it to be like 10 new flows for 10 TCP > > sessions? I see this counter increase by the thousands per second. > > This could be normal. When a flow queue is completely drained, it will > be deleted. Next packet will be in the "new_flows". This is another > sign of the bottleneck maybe at TCP stack. Right, Toke pointed that out too. > > CPU load is not an issue AFAICT, even with all the debugging being > > written into the syslog (or journal or something) that's the only thing > > that takes noticable CPU time - ~50% for systemd-journal and ~20% for > > rsyslogd, <10% for the throughput testing program and that's about it. > > The system has 4 threads and seems mostly idle. > > What's CPU usage for software irq? Is CPU usage average of all cores? > Maybe the core that handles packet processing or softirq is maxed out > even the average is fine. Agree, I didn't really say that well. I'm really just ballparking this by using 'top', but even the *most* loaded of the 4 CPUs (threads?) is at >80% idle, <6% softirq and <12% sys for the duration of the test. > Are you using iperf or netperf? increase the TCP windows size may > help. Adjust things like "net.core.wmem_max" and "net.ipv4.tcp_mem" > maybe necessary to enable iperf to use larger windows. Chariot :) Anyway, this had no effect. I'll go play with the TSO next, for some reason our driver won't let me disable it dynamically (I guess I should just fix that). johannes
