Re: [RFC PATCH] ARM: smp: Fix the CPU hotplug race with scheduler.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 6/20/2011 7:53 PM, Russell King - ARM Linux wrote:
On Mon, Jun 20, 2011 at 12:40:19PM +0100, Russell King - ARM Linux wrote:
Ok.  So loops_per_jiffy must be too small.  My guess is you're using an
older kernel without 71c696b1 (calibrate: extract fall-back calculation
into own helper).

Right, this commit above helps show the problem - and it's fairly subtle.

It's a race condition.  Let's first look at the spinlock debugging code.
It does this:

static void __spin_lock_debug(raw_spinlock_t *lock)
{
         u64 i;
         u64 loops = loops_per_jiffy * HZ;

         for (;;) {
                 for (i = 0; i<  loops; i++) {
                         if (arch_spin_trylock(&lock->raw_lock))
                                 return;
                         __delay(1);
                 }
		/* print warning */
	}
}

If loops_per_jiffy is zero, we never try to grab the spinlock, because
we never enter the inner for loop.  We immediately print a warning,
and re-execute the outer loop for ever, resulting in the CPU locking up
in this condition.

In theory, we should never see a zero loops_per_jiffy value, because it
represents the number of loops __delay() needs to delay by one jiffy and
clearly zero makes no sense.

However, calibrate_delay() does this (which x86 and ARM call on secondary
CPU startup):

calibrate_delay()
{
...
	if (preset_lpj) {
	} else if ((!printed)&&  lpj_fine) {
	} else if ((loops_per_jiffy = calibrate_delay_direct()) != 0) {
	} else {
		/* approximation/convergence stuff */
	}
}

Now, before 71c696b, this used to be:

         } else {
                 loops_per_jiffy = (1<<12);

So the window between calibrate_delay_direct() returning and setting
loops_per_jiffy to zero, and the re-initialization of loops_per_jiffy
was relatively short (maybe even the compiler optimized away the zero
write.)

However, after 71c696b, this now does:

         } else {
                 if (!printed)
                         pr_info("Calibrating delay loop... ");
+               loops_per_jiffy = calibrate_delay_converge();

So, as loops_per_jiffy is not local to this function, the compiler has
to write out that zero value, before calling calibrate_delay_converge(),
and loops_per_jiffy only becomes non-zero _after_ calibrate_delay_converge()
has returned.  This opens the window and allows the spinlock debugging
code to explode.

This patch closes the window completely, by only writing to loops_per_jiffy
only when we have a real value for it.

This allows me to boot 3.0.0-rc3 on Versatile Express (4 CPU) whereas
without this it fails with spinlock lockup and rcu problems.

  init/calibrate.c |   14 ++++++++------
  1 files changed, 8 insertions(+), 6 deletions(-)

I am away from my board now. Will test this change.
btw, the online-active race is still open even with this patch close
and should be fixed.

Regards
Santosh

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Arm (vger)]     [ARM Kernel]     [ARM MSM]     [Linux Tegra]     [Linux WPAN Networking]     [Linux Wireless Networking]     [Maemo Users]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite Trails]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux