On 3 February 2015 at 15:27, Eric Dumazet <eric.dumazet@xxxxxxxxx> wrote: > On Tue, 2015-02-03 at 12:50 +0100, Michal Kazior wrote: [...] >> IOW: >> - stretch acks / TSO defer don't seem to help much (when compared to >> throughput results from yesterday) >> - GRO helps >> - disabling A-MSDU on sender helps >> - net/master+GRO still doesn't reach the performance from before the >> regression (~600mbps w/ GRO) >> >> You can grab logs and dumps here: http://www.filedropper.com/test2tar >> > > Thanks for these traces. > > There is absolutely a problem at the sender, as we can see a big 2ms > delay between reception of ACK and send of following packets. > TCP stack should generate them immediately. > Are you using some kind of netem qdisc ? Both systems have identical setup: ; tc qdisc qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc mq 0: dev wlan1 root qdisc pfifo_fast 0: dev wlan1 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan1 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan1 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan1 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 > These 2ms delays, in a flow with a 5ms RTT are terrible. > > 06:54:57.408391 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294899240, win 11268, options [nop,nop,TS val 1053302 ecr 1052250], length 0 > 06:54:57.408418 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294910824, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > 06:54:57.408431 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294936888, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > 06:54:57.408453 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 4294962952, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > 06:54:57.408474 IP 192.168.1.2.5001 > 192.168.1.3.51645: Flags [.], ack 0, win 11268, options [nop,nop,TS val 1053303 ecr 1052251], length 0 > <this 2ms delay is not generated by TCP stack.> > 06:54:57.410243 IP 192.168.1.3.51645 > 192.168.1.2.5001: Flags [.], seq 82536:83984, ack 1, win 457, options [nop,nop,TS val 1052256 ecr 1053303], length 1448 [...] > > Are packets TX completed after a timer or something ? As far as ath10k is concerned - no timers here. Not sure about firmware itself though. > Some very heavy stuff might run from tasklet (or other softirq triggered) event. > > BTW, traces tend to show that you 'receive' multiple ACK in the same burst, > its not clear if they are delayed at one side or the other. > > GRO should delay only GRO candidates. ACK packets are not GRO candidates. > > Have you tried to disable GSO on sender ? I assume I do that via ethtool? This is my current setup on both systems: ; ethtool -k wlan1 Features for wlan1: rx-checksumming: off [fixed] tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off [fixed] tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: on [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] ; ethtool -K wlan1 generic-segmentation-offload off ethtool: bad command line argument(s) For more information run ethtool -h > (Or maybe wifi drivers should start to use skb->xmit_more as a signal to end aggregation) This could work if your firmware/device supports this kind of thing. To my understanding ath10k firmware doesn't. Michał -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
