with a running system, can you please tell me that how to check whatthe driver model is, NAPI or interrupt? 2008/10/15, Madhukar.Mythri@xxxxxxxxxxx <Madhukar.Mythri@xxxxxxxxxxx>:>> First make-sure whether you have one dedicated interrupt-line for each> network card.> You got 100% CPU utilization with NAPI Disabled or Enabled ?>> With NAPI enabled, and assign each interrupt-line to each CPU. And check> whether the interrupts are distributed as per you assigned, by seeing> "cat /proc/interrupts" and by pumping traffic.> And also check how the interrupts balanced between CPU's.>> As per my observation, when NAPI enabled, the interrupt-line cannot> assign to each CPU. In NAPI-mode for whichever the CPU got the first> interrupt, that CPU only carried out in poll-mode for rest of the loaded> packets(H/W buffers).>> So, with NAPI enabled, if interrupts are not distributed on assigned> CPU's and one CPU got loaded more, then try by Disabling NAPI, and use> 'irqbalance' utility to load-balance the interrupts between all CPU's.> (http://www.irqbalance.org/).>> Regards,> Madhukar.>> -----Original Message-----> From: Don Porter [mailto:porterde@xxxxxxxxxxxxx]> Sent: Wednesday, October 15, 2008 6:23 PM> To: Mythri, Madhukar [NETPWR/EMBED/HYDE]> Cc: linux-net@xxxxxxxxxxxxxxx> Subject: Re: e1000 softirq load balancing>> I believe I have 4 interrupt lines, but I will have to double-check.>> I have successfully used /proc/irq/####/smp_affinity to assign the> interrupts to 4 different CPUs. The problem is in the softirq portion> of the interrupt handling, where CPU usage indicates that they are all> being funneled back to a single CPU.>> Does that make sense? I feel like one ought to be able to have 4> softirq daemons servicing incoming packets, not just one.>> Thanks,> Don>> Madhukar.Mythri@xxxxxxxxxxx wrote:> > You are saying that, 4 Intel 82571EB Gb NICs (2 pci cards x 2> > NICs/chip) using the e1000 driver.> > So, do you have 4 interrupt lines or only 2-lines?> >> > Based on this, if you have 4 interrupt lines, then you can assign each>> > interrupt-line to each CPU-core, with NAPI enabled(for good> > performance).> >> > Regards,> > Madhukar.> > -----Original Message-----> > From: linux-net-owner@xxxxxxxxxxxxxxx> > [mailto:linux-net-owner@xxxxxxxxxxxxxxx] On Behalf Of Don Porter> > Sent: Wednesday, October 15, 2008 12:36 AM> > To: linux-net@xxxxxxxxxxxxxxx> > Subject: e1000 softirq load balancing> >> > Hi,> >> > Background:> >> > I have a 16 core x86_64 machine (4 chips x 4 cores/chip) that has 4> > Intel 82571EB Gb NICs (2 pci cards x 2 NICs/chip) using the e1000> > driver.> >> > I have a simple client/server micro-benchmark that pounds a server on> > each NIC with requests to measure peak throughput. I am running> > Ubuntu 8.04.1, kernel version 2.6.24.> >> > Problem:> >> > What I am observing is that a single ksoftirqd thread is becoming a> > bottleneck for the system.> > More specifically, one cpu runs ksoftirqd at 100% cpu utilization,> > while> > 4 cpus each run their servers at about 25%. I carefully used> > sched_setaffinity() to map server threads to cpus and> > /proc/irq/<device>/smp_affinity to map hardware interrupts to cpus> > such that there should be exactly 1 cpu per server thread and 1 cpu> > for servicing hardware interrupts per device.> >> > I can observe (via /proc/interrupts) that the interrupts are being> > distributed properly, but despite this I only see 1 or 2 ksoftirqd> > running, and the server daemons bottlenecked behind them. (This is> > with NAPI disabled. With NAPI enabled, I can't get even 2 ksoftirqd> > threads to run). I have tried varous permutations such as assigning> > each hardware interrupt to a different physical chip.> >> > Desired Result:> >> > It seems to me that with 4 independent NICs and plenty of CPUs to> > spare, I ought to be able to assign one softirq daemon to each NIC> > rather than funnelling all of the traffic through 1 or 2.> >> > Any advice on this issue is greatly appreciated.> >> > Best regards,> > Don Porter> > --> > To unsubscribe from this list: send the line "unsubscribe linux-net"> > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo> > info at http://vger.kernel.org/majordomo-info.html> >>> --> To unsubscribe from this list: send the line "unsubscribe linux-net" in> the body of a message to majordomo@xxxxxxxxxxxxxxx> More majordomo info at http://vger.kernel.org/majordomo-info.html> -- Best Regards, 薛 勇 QQ:312200 e-mail:ultraice@xxxxxxxxxxxx:it@xxxxxxxxxxxxxxxxx?韬{.n?????%??檩??w?{.n???{炳w?)?骅w* jg????????G??⒏ ⒎?:+v???????? ????＂??????
