On Tue, 13 Nov 2007 10:12:03 +0100 Marek Kierdelewicz <marek@xxxxxxxxx> wrote: > >Hi All, > > Hi, > > > I have a box with 24 e1000 cards in it. They are configured as 12 > >bridges, each with 2 ports. > > 24 ports of e1000 nics means 24 interrupts used (or shared). Maybe > thats the source of the problem. Did you notice anything unusual in your > logs concerning e1000 nics? > > >... > >CPU utilisation is hovering around 50%, and load average is > >consistently > >under 0.1, so I don't beleive I'm looking at a CPU bottleneck. > > Is your box is multi-core (or HT-enabled)? Is your kernel SMP? If thats > the case then check per core CPU utilisation (press "1" when watching > top). You may be hitting roof only on one of the cores while avg. > utilisation is around 50%. If you're not familiar with "smp_affinity", > then you should read the following: > http://bcr2.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt > > cheers, > Marek Kierdelewicz > KoBa ISP > _______________________________________________ > Bridge mailing list > Bridge@xxxxxxxxxxxxxxxxxxxxxxxxxx > https://lists.linux-foundation.org/mailman/listinfo/bridge First, make sure you have enough bus bandwidth! What kind of box, you really need PCI-express to get better bus throughput. MSI will also help. Memory speeds also matter. And the dual-port e1000's add a layer of PCI bridge that also hurts latency/bandwidth. Don't use kernel irq balancing, user space irqbalance daemon is smart enough to recognize network device's and do the right thing (assign them directly to processors). It would be useful to see what the kernel profiling (oprofile) shows. How are you measuring CPU utilization? The only accurate way is to measure time with an idle soaker program versus, time under load. Andrew Morton wrote a cyclesoaker to do this, if you want it, I'll dig it up. -- Stephen Hemminger <shemminger@xxxxxxxxxxxxxxxxxxxx> _______________________________________________ Bridge mailing list Bridge@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linux-foundation.org/mailman/listinfo/bridge
