Thanks Eric to provide the data. I am looping Tom (as I am looking into his recent patches) and Olaf (from Suse). So, if I understand it correctly, you are running netperf with single TCP connection, and you got ~26Gbps initially and got ~30Gbps after turning the tx-usecs and tx-frames. Do you have a baseline on your environment for the best/max/or peak throughput? Again, in my environment (SLES bare metal), if use SLES12 default kernel as a baseline, we can see significant performance drop (10% ~ 50%) on latest linux-next kernel. Absolutely I will try the same test on net-next soon and update the results to here later. Thanks, Simon > -----Original Message----- > From: Eric Dumazet [mailto:eric.dumazet@xxxxxxxxx] > Sent: Saturday, November 7, 2015 11:50 AM > To: David Ahern <dsa@xxxxxxxxxxxxxxxxxxx> > Cc: Simon Xiao <sixiao@xxxxxxxxxxxxx>; devel@xxxxxxxxxxxxxxxxxxxxxx; > netdev@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; David Miller > <davem@xxxxxxxxxxxxx>; KY Srinivasan <kys@xxxxxxxxxxxxx>; Haiyang > Zhang <haiyangz@xxxxxxxxxxxxx> > Subject: Re: linux-next network throughput performance regression > > On Sat, 2015-11-07 at 11:35 -0800, Eric Dumazet wrote: > > On Fri, 2015-11-06 at 14:30 -0700, David Ahern wrote: > > > On 11/6/15 2:18 PM, Simon Xiao wrote: > > > > The .config file used to build linux-next kernel is attached to this mail. > > > > > > Thanks. > > > > > > Failed to notice this on the first response; my brain filled in. Why > > > linux-next tree? Can you try net-next which is more relevant for > > > this mailing list, post the top commit id and config file used? > > > > Throughput on a single TCP flow for a 40G NIC can be tricky to tune. > > > > Make sure IRQ are properly setup/balanced, as I know that IRQ names > > were changed recently and your scripts might have not noticed... > > > > Also "ethtool -c eth0" might show very different interrupt coalescing > > params ? > > > > I too have a Mellanox 40Gb in my lab and saw no difference in > > performance with recent kernels. > > > > Of course, a simple "perf record -a -g sleep 4 ; perf report" might > > point to some obvious issue. Like unexpected segmentation in case of > > forwarding... > > > > > > I did a test with current net tree on both sender and receiver > > lpaa23:~# ./netperf -H 10.246.7.152 > MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > 10.246.7.152 () port 0 AF_INET > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 87380 16384 16384 10.00 26864.98 > lpaa23:~# ethtool -c eth1 > Coalesce parameters for eth1: > Adaptive RX: on TX: off > stats-block-usecs: 0 > sample-interval: 0 > pkt-rate-low: 400000 > pkt-rate-high: 450000 > > rx-usecs: 16 > rx-frames: 44 > rx-usecs-irq: 0 > rx-frames-irq: 0 > > tx-usecs: 16 > tx-frames: 16 > tx-usecs-irq: 0 > tx-frames-irq: 256 > > rx-usecs-low: 0 > rx-frame-low: 0 > tx-usecs-low: 0 > tx-frame-low: 0 > > rx-usecs-high: 128 > rx-frame-high: 0 > tx-usecs-high: 0 > tx-frame-high: 0 > > lpaa23:~# ethtool -C eth1 tx-usecs 4 tx-frames 4 > lpaa23:~# ./netperf -H > 10.246.7.152 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 > AF_INET to > 10.246.7.152 () port 0 AF_INET > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 87380 16384 16384 10.00 30206.27 > _______________________________________________ devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxx http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
