There is a dead lock between between cpu hotplug and handler of ACPI Processor
Power notify event. Cpu hotplug lock is held and then hold cpuidle lock during cpu hotplug.
During ACPI processor power notify event, acpi_processor_cst_has_chaged() does converse
thing and cause dead lock. Just like the following log shows.
[ 91.456670] ======================================================
[ 91.457160] [ INFO: possible circular locking dependency detected ]
[ 91.457658] 3.17.0-rc3+ #15 Not tainted
[ 91.457968] -------------------------------------------------------
[ 91.458463] kworker/0:1/75 is trying to acquire lock:
[ 91.458866] (cpu_hotplug.lock){++++++}, at: [<ffffffff810c3334>] get_online_cpus+0x24/0x70
[ 91.459600]
[ 91.459600] but task is already holding lock:
[ 91.460062] (cpuidle_lock){+.+.+.}, at: [<ffffffff81666147>] cpuidle_pause_and_lock+0x17/0x40
[ 91.460810]
[ 91.460810] which lock already depends on the new lock.
[ 91.460810]
[ 91.461451]
[ 91.461451] the existing dependency chain (in reverse order) is:
[ 91.462038]
[ 91.462038] -> #2 (cpuidle_lock){+.+.+.}:
[ 91.462442] [<ffffffff8110cba4>] lock_acquire+0xa4/0x120
[ 91.462931] [<ffffffff817eb4de>] mutex_lock_nested+0x4e/0x3b0
[ 91.463453] [<ffffffff81666147>] cpuidle_pause_and_lock+0x17/0x40
[ 91.464000] [<ffffffff8148ea5b>] acpi_processor_hotplug+0x49/0x8d
[ 91.464548] [<ffffffff8148cc9f>] acpi_cpu_soft_notify+0xaf/0xe4
[ 91.465081] [<ffffffff810e20ac>] notifier_call_chain+0x4c/0x70
[ 91.465609] [<ffffffff810e217e>] __raw_notifier_call_chain+0xe/0x10
[ 91.466169] [<ffffffff810c3443>] cpu_notify+0x23/0x50
[ 91.466637] [<ffffffff810c36a0>] _cpu_up+0x150/0x160
[ 91.467098] [<ffffffff817dba6f>] enable_nonboot_cpus+0xaf/0x170
[ 91.467632] [<ffffffff81113085>] hibernation_snapshot+0x275/0x380
[ 91.468181] [<ffffffff81113990>] hibernate+0x160/0x210
[ 91.468655] [<ffffffff81111314>] state_store+0xe4/0xf0
[ 91.469129] [<ffffffff813eb16f>] kobj_attr_store+0xf/0x20
[ 91.469624] [<ffffffff812aba54>] sysfs_kf_write+0x44/0x60
[ 91.470118] [<ffffffff812ab357>] kernfs_fop_write+0xe7/0x170
[ 91.470632] [<ffffffff81230da7>] vfs_write+0xb7/0x1f0
[ 91.471100] [<ffffffff81231959>] SyS_write+0x49/0xb0
[ 91.471561] [<ffffffff817ee429>] system_call_fastpath+0x16/0x1b
[ 91.472096]
[ 91.472096] -> #1 (cpu_hotplug.lock#2){+.+.+.}:
[ 91.472556] [<ffffffff8110cba4>] lock_acquire+0xa4/0x120
[ 91.473044] [<ffffffff817eb4de>] mutex_lock_nested+0x4e/0x3b0
[ 91.473564] [<ffffffff810c34df>] cpu_hotplug_begin+0x4f/0x80
[ 91.474078] [<ffffffff810c3584>] _cpu_up+0x34/0x160
[ 91.474532] [<ffffffff810c3709>] cpu_up+0x59/0x80
[ 91.474973] [<ffffffff81d8170c>] smp_init+0x86/0x88
[ 91.475429] [<ffffffff81d5e203>] kernel_init_freeable+0x16c/0x27b
[ 91.475976] [<ffffffff817da77e>] kernel_init+0xe/0xf0
[ 91.476443] [<ffffffff817ee37c>] ret_from_fork+0x7c/0xb0
[ 91.476930]
[ 91.476930] -> #0 (cpu_hotplug.lock){++++++}:
[ 91.477359] [<ffffffff8110c0b0>] __lock_acquire+0x1760/0x1a70
[ 91.477880] [<ffffffff8110cba4>] lock_acquire+0xa4/0x120
[ 91.478367] [<ffffffff810c335a>] get_online_cpus+0x4a/0x70
[ 91.478868] [<ffffffff8148eb12>] acpi_processor_cst_has_changed+0x73/0x17d
[ 91.479475] [<ffffffff8148cd5e>] acpi_processor_notify+0x8a/0xe2
[ 91.480014] [<ffffffff814743fb>] acpi_ev_notify_dispatch+0x44/0x5c
[ 91.480569] [<ffffffff8146019a>] acpi_os_execute_deferred+0x14/0x20
[ 91.481130] [<ffffffff810dba1f>] process_one_work+0x1df/0x4d0
[ 91.481652] [<ffffffff810dbe2b>] worker_thread+0x11b/0x490
[ 91.482152] [<ffffffff810e100d>] kthread+0xed/0x110
[ 91.482606] [<ffffffff817ee37c>] ret_from_fork+0x7c/0xb0
[ 91.483094]
[ 91.483094] other info that might help us debug this:
[ 91.483094]
[ 91.483721] Chain exists of:
[ 91.483721] cpu_hotplug.lock --> cpu_hotplug.lock#2 --> cpuidle_lock
[ 91.483721]
[ 91.484486] Possible unsafe locking scenario:
[ 91.484486]
[ 91.491113] CPU0 CPU1
[ 91.494496] ---- ----
[ 91.497833] lock(cpuidle_lock);
[ 91.501083] lock(cpu_hotplug.lock#2);
[ 91.504572] lock(cpuidle_lock);
[ 91.507944] lock(cpu_hotplug.lock);
[ 91.511115]
[ 91.511115] *** DEADLOCK ***
[ 91.511115]
[ 91.519889] 3 locks held by kworker/0:1/75:
[ 91.522918] #0: ("kacpi_notify"){++++..}, at: [<ffffffff810db9bd>] process_one_work+0x17d/0x4d0
[ 91.526445] #1: ((&dpc->work)){+.+...}, at: [<ffffffff810db9bd>] process_one_work+0x17d/0x4d0
[ 91.530031] #2: (cpuidle_lock){+.+.+.}, at: [<ffffffff81666147>] cpuidle_pause_and_lock+0x17/0x40
[ 91.533653]
[ 91.533653] stack backtrace:
[ 91.539712] CPU: 0 PID: 75 Comm: kworker/0:1 Not tainted 3.17.0-rc3+ #15
[ 91.543086] Hardware name: Intel Corporation CHERRYVIEW A3 PLATFORM/Braswell CRB, BIOS BRASWEL1.X64.0035.R00.1409041111 09/04/2014
[ 91.546930] Workqueue: kacpi_notify acpi_os_execute_deferred
[ 91.550405] ffffffff82656d40 ffff880078883b68 ffffffff817e44fd ffffffff82656b90
[ 91.554096] ffff880078883ba8 ffffffff817dff2a ffff880078883c00 ffff88007884e218
[ 91.557807] 0000000000000002 0000000000000003 ffff88007884e218 ffff88007884da50
[ 91.561524] Call Trace:
[ 91.564833] [<ffffffff817e44fd>] dump_stack+0x45/0x56
[ 91.568350] [<ffffffff817dff2a>] print_circular_bug+0x200/0x20f
[ 91.571946] [<ffffffff8110c0b0>] __lock_acquire+0x1760/0x1a70
[ 91.575529] [<ffffffff8110a42a>] ? mark_held_locks+0x6a/0x90
[ 91.579100] [<ffffffff810856f9>] ? flat_send_IPI_allbutself+0xe9/0x140
[ 91.582747] [<ffffffff8110cba4>] lock_acquire+0xa4/0x120
[ 91.586323] [<ffffffff810c3334>] ? get_online_cpus+0x24/0x70
[ 91.589927] [<ffffffff810c335a>] get_online_cpus+0x4a/0x70
[ 91.593520] [<ffffffff810c3334>] ? get_online_cpus+0x24/0x70
[ 91.597125] [<ffffffff8148eb12>] acpi_processor_cst_has_changed+0x73/0x17d
[ 91.600840] [<ffffffff8148cd5e>] acpi_processor_notify+0x8a/0xe2
[ 91.604508] [<ffffffff814743fb>] acpi_ev_notify_dispatch+0x44/0x5c
[ 91.608188] [<ffffffff8146019a>] acpi_os_execute_deferred+0x14/0x20
[ 91.611856] [<ffffffff810dba1f>] process_one_work+0x1df/0x4d0
[ 91.615495] [<ffffffff810db9bd>] ? process_one_work+0x17d/0x4d0
[ 91.619121] [<ffffffff810dbe2b>] worker_thread+0x11b/0x490
[ 91.622713] [<ffffffff810dbd10>] ? process_one_work+0x4d0/0x4d0
[ 91.626347] [<ffffffff810e100d>] kthread+0xed/0x110
[ 91.629907] [<ffffffff810e0f20>] ? kthread_create_on_node+0x200/0x200
[ 91.633606] [<ffffffff817ee37c>] ret_from_fork+0x7c/0xb0
[ 91.637238] [<ffffffff810e0f20>] ? kthread_create_on_node+0x200/0x200
This patch is to change the sequence of holding cpu hotplug
and cpu idle lock in the acpi_processor_cst_has_changed() to avoid the
dead lock.
Signed-off-by: Lan Tianyu <tianyu.lan@xxxxxxxxx>
---
drivers/acpi/processor_idle.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c
index 3dca36d..385ec5f 100644
--- a/drivers/acpi/processor_idle.c
+++ b/drivers/acpi/processor_idle.c
@@ -1071,9 +1071,9 @@ int acpi_processor_cst_has_changed(struct acpi_processor *pr)
if (pr->id == 0 && cpuidle_get_driver() == &acpi_idle_driver) {
- cpuidle_pause_and_lock();
/* Protect against cpu-hotplug */
get_online_cpus();
+ cpuidle_pause_and_lock();
/* Disable all cpuidle devices */
for_each_online_cpu(cpu) {
@@ -1100,8 +1100,9 @@ int acpi_processor_cst_has_changed(struct acpi_processor *pr)
cpuidle_enable_device(dev);
}
}
- put_online_cpus();
+
cpuidle_resume_and_unlock();
+ put_online_cpus();
}
return 0;
--
1.8.4.rc0.1.g8f6a3e5.dirty
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
