You are saying that, 4 Intel 82571EB Gb NICs (2 pci cards x 2 NICs/chip) using the e1000 driver. So, do you have 4 interrupt lines or only 2-lines? Based on this, if you have 4 interrupt lines, then you can assign each interrupt-line to each CPU-core, with NAPI enabled(for good performance). Regards, Madhukar. -----Original Message----- From: linux-net-owner@xxxxxxxxxxxxxxx [mailto:linux-net-owner@xxxxxxxxxxxxxxx] On Behalf Of Don Porter Sent: Wednesday, October 15, 2008 12:36 AM To: linux-net@xxxxxxxxxxxxxxx Subject: e1000 softirq load balancing Hi, Background: I have a 16 core x86_64 machine (4 chips x 4 cores/chip) that has 4 Intel 82571EB Gb NICs (2 pci cards x 2 NICs/chip) using the e1000 driver. I have a simple client/server micro-benchmark that pounds a server on each NIC with requests to measure peak throughput. I am running Ubuntu 8.04.1, kernel version 2.6.24. Problem: What I am observing is that a single ksoftirqd thread is becoming a bottleneck for the system. More specifically, one cpu runs ksoftirqd at 100% cpu utilization, while 4 cpus each run their servers at about 25%. I carefully used sched_setaffinity() to map server threads to cpus and /proc/irq/<device>/smp_affinity to map hardware interrupts to cpus such that there should be exactly 1 cpu per server thread and 1 cpu for servicing hardware interrupts per device. I can observe (via /proc/interrupts) that the interrupts are being distributed properly, but despite this I only see 1 or 2 ksoftirqd running, and the server daemons bottlenecked behind them. (This is with NAPI disabled. With NAPI enabled, I can't get even 2 ksoftirqd threads to run). I have tried varous permutations such as assigning each hardware interrupt to a different physical chip. Desired Result: It seems to me that with 4 independent NICs and plenty of CPUs to spare, I ought to be able to assign one softirq daemon to each NIC rather than funnelling all of the traffic through 1 or 2. Any advice on this issue is greatly appreciated. Best regards, Don Porter -- To unsubscribe from this list: send the line "unsubscribe linux-net" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-net" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
