Hi,
In the 4.19/5.0-rt configured !SMP, the following chain of events can
take place:
------------------------------ %< -------------------------------------
smp_apic_timer_interrupt(){
hrtimer_interrupt() {
__hrtimer_run_queues() {
watchdog_timer_fn() {
stop_one_cpu_nowait() {
#ifdef !CONFIG_SMP
schedule_work() {
queue_work() {
queue_work_on() {
/* phew, long, ah!?.... */
local_lock_irqsave() {
__local_lock_irqsave() {
__local_lock_irq() {
spin_lock_irqsave() {
rt_spin_lock()...
/* from here on you already know...*/
/* a lot of } */
------------------------------ >% -------------------------------------
And this can cause a scheduling while in atomic.
Tracking down, the commit that introduced the stop_one_cpu_nowait()->
queue_work_on() to the path was:
------------------------------ %< -------------------------------------
commit 9cf57731b63e37ed995b46690adc604891a9a28f
Author: Peter Zijlstra
Date: Thu Jun 7 10:52:03 2018 +0200
watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
Oleg suggested to replace the "watchdog/%u" threads with
cpu_stop_work. That removes one thread per CPU while at the same time
fixes softlockup vs SCHED_DEADLINE.
But more importantly, it does away with the single
smpboot_update_cpumask_percpu_thread() user, which allows
cleanups/shrinkage of the smpboot interface.
Suggested-by: Oleg Nesterov
Signed-off-by: Peter Zijlstra (Intel)
Cc: Linus Torvalds
Cc: Peter Zijlstra
Cc: Thomas Gleixner
Cc: linux-kernel@xxxxxxxxxxxxxxx
Signed-off-by: Ingo Molnar
------------------------------ >% -------------------------------------
Later modified by:
------------------------------ %< -------------------------------------
commit be45bf5395e0886a93fc816bbe41a008ec2e42e2
Author: Peter Zijlstra
Date: Fri Jul 13 12:42:08 2018 +0200
watchdog/softlockup: Fix cpu_stop_queue_work() double-queue bug
When scheduling is delayed for longer than the softlockup interrupt
period it is possible to double-queue the cpu_stop_work, causing list
corruption.
Cure this by adding a completion to track the cpu_stop_work's
progress.
Reported-by: kernel test robot
Tested-by: Rong Chen
Signed-off-by: Peter Zijlstra (Intel)
Cc: Linus Torvalds
Cc: Peter Zijlstra
Cc: Thomas Gleixner
Fixes: 9cf57731b63e ("watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work")
Link: http://lkml.kernel.org/r/20180713104208.GW2494@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Signed-off-by: Ingo Molnar
------------------------------ >% -------------------------------------
I hit it while trying the automata model against the 4.19-rt/5.0-rt
kernels.
You can find more about it here:
http://bristot.me/another-bug-found-with-the-model/
Thanks
-- Daniel
