Re: Nehalem network performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, 2010-01-27 at 10:49 -0500, Kelvin Ku wrote:
> > 
> > Please post the output of:
> > $ cat /proc/interrupts | grep eth
> 
> We rename our interfaces to lan:
> 
> $ grep lan /proc/interrupts 
>  61:          1          0          0          0   PCI-MSI-edge      lan0
>  62:    7194004          0          0          0   PCI-MSI-edge      lan0-TxRx-0
>  63:          0          1          0          0   PCI-MSI-edge      lan1
>  64:          0          0   49842410          0   PCI-MSI-edge      lan1-TxRx-0
> 
> Note that irqbalance is disabled. I found that it wasn't balancing IRQs like on
> our older machines. I note that the irqbalance docs say that NIC interrupts
> should not be balanced, which is what we're seeing whether irqbalance is running or not.

Interrupts look OK. (You want the interrupts for each MSI queue to land
in one CPU core instead of getting bumped around.


> lan1 (multicast interface) is below. Note that rx_missed_errors is non-zero. I
> previously encountered this with the e1000e NIC after disabling cpuspeed, 

I usually disable cpuspeed, but my boxes have multiple 1 and 10GbE
NICs...


> which
> was throttling the CPUs to 1.6 GHz (from a maximum of 2.4 GHz). I attempted to
> remedy this by setting InterruptThrottleRate=0,0 in the e1000e driver, after
> which we had one full day of testing with zero rx_missed_errors, but the
> application still reported packet loss.

rx_missed_error usually get triggered when the kernel is slow to handle
incoming hardware interrupts.
There's a trade-off here, increase the interrupt rate and you'll
increase the kernel CPU usage as the expense of lower latency - decrease
the interrupt rate, and you'll reduce the CPU usage at the expense of a
higher chance of hitting the RX queue limit.
I'd suggest you try setting the InterruptThrottleRate to 1000, while
increasing the RX queues to 4096.
(sbin/ethtool -G DEVICE rx 4096)

You could try enabling multi-queue by adding IntterruptType=2,
RSS=NUM_OF_QUEUE and MQ=1 to your modprobe.conf.d.

> Supermicro X8DTL-iF

We are using a similar SuperMicro board for 10GbE.

> > Have you tried enabling pci=msi in your kernel's command line?
> 
> No. Do I need to do this? MSI seems to be enabled:

OK. looks OK.

> Agreed. I ran a local netperf test and was seeing about 8 Gbps of throughput on a single core, so this should be adequate for 1 Gbps traffic.

Can you post the output of $ mpstat -P 1 ALL during peak load?

- Gilboa

-- 
users mailing list
users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [Fedora Magazine]     [Fedora News]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Maintainers]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [Fedora Fonts]     [ATA RAID]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [SSH]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Tux]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Asterisk PBX]     [Fedora Sparc]     [Fedora Universal Network Connector]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux