Allow pending IPIs to interrupt a timer interrupt that is looping in the do_timer() "while" loop in timer_interrupt(). (Interrupts are allowed at only 1 spot in the code). Signed-off-by: Jack Steiner <steiner@xxxxxxx> --- We have seen isolated cases where a cpu fails to respond to an IPI for an extended period of time. This has only been seen on very large (1024p) HT-enabled montecito systems running workloads that cause EXTREME contention on the BKL lock. One failing senario appears to be: - Lots of cpus already contending on BKL - One cpu runs the haldaemon. This daemon grabs the BKL, then sends an IPI to all other cpus (see invalidate_bh_lrus()). - Cpus that are already processing timer ticks don't respond to the IPI until it exits from the timer interrupt exception - The haldaemon will hold the BKL until all cpus have responded to the IPI. This increases or prolongs the BKL contention. - A cpu may decide to do a global load_balance as part of timer tick processing. - A global load_balance requires lots of off-node memory references - Off node memory references are really slow (not sure if this is a chipset or montecito issue) during periods of SEVERE(!!!) lock contention. - If the global load_balance takes > 4 msec, the while loop is reexecuted. This can continue for multiple iterations of the loop. IPIs will not be processed until the loops completes & timer_interrupt() exits. There are probably other failing senarios. We have a workload that causes system lockups within 5 - 10 minutes running the standard kernel. With this patch, we have had 2 overnight runs with no failures. I still want to identify the root cause of the poor performance but this patch appears to work around the problem. The patch should have no impact on smaller systems - they are unlikely to loop very often in the "while" loop. No new code is executed unless the loop is reexecuted. Index: linux/arch/ia64/kernel/time.c =================================================================== --- linux.orig/arch/ia64/kernel/time.c 2006-10-16 10:53:45.000000000 -0500 +++ linux/arch/ia64/kernel/time.c 2006-10-16 12:14:23.536140299 -0500 @@ -84,6 +84,12 @@ timer_interrupt (int irq, void *dev_id) if (time_after(new_itm, ia64_get_itc())) break; + + /* + * Allow IPIs to interrupt the timer loop. + */ + local_irq_enable(); + local_irq_disable(); } do { - To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
![]() |