> On Jan 1, 2023, at 8:02 AM, Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> wrote: > > On Sun, Jan 1, 2023 at 2:16 PM Joel Fernandes (Google) > <joel@xxxxxxxxxxxxxxxxx> wrote: >> >> During shutdown of rcutorture, the shutdown thread in >> rcu_torture_cleanup() calls torture_cleanup_begin() which sets fullstop >> to FULLSTOP_RMMOD. This is enough to cause the rcutorture threads for >> readers and fakewriters to breakout of their main while loop and start >> shutting down. >> >> Once out of their main loop, they then call torture_kthread_stopping() >> which in turn waits for kthread_stop() to be called, however >> rcu_torture_cleanup() has not even called kthread_stop() on those >> threads yet, it does that a bit later. However, before it gets a chance >> to do so, torture_kthread_stopping() calls >> schedule_timeout_interruptible(1) in a tight loop. Tracing confirmed >> this makes the timer softirq constantly execute timer callbacks, while >> never returning back to the softirq exit path and is essentially "locked >> up" because of that. If the softirq preempts the shutdown thread, >> kthread_stop() may never be called. >> >> This commit improves the situation dramatically, by increasing timeout >> passed to schedule_timeout_interruptible() 1/20th of a second. This >> causes the timer softirq to not lock up a CPU and everything works fine. >> Testing has shown 100 runs of TREE07 passing reliably, which was not the >> case before because of RCU stalls. > On my Dell PowerEdge R720 with two Intel(R) Xeon(R) CPU E5-2660 128G memory: > 1) before this patch: > 3 of 80 rounds failed with "rcu: INFO: rcu_sched detected stalls on > CPUs/tasks" [1] > 2) after this patch > all 80 rounds passed > > Tested-by: Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> > Thanks! Glad to see your tests look good now. - Joel > Thanks > Zhouyi > > [1] http://154.220.3.115/logs/20230101/console.log >> >> Cc: Paul McKenney <paulmck@xxxxxxxxxx> >> Cc: Frederic Weisbecker <fweisbec@xxxxxxxxx> >> Cc: Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> >> Cc: <stable@xxxxxxxxxxxxxxx> # 6.0.x >> Signed-off-by: Joel Fernandes (Google) <joel@xxxxxxxxxxxxxxxxx> >> --- >> kernel/torture.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/kernel/torture.c b/kernel/torture.c >> index 29afc62f2bfe..d024f3b7181f 100644 >> --- a/kernel/torture.c >> +++ b/kernel/torture.c >> @@ -915,7 +915,7 @@ void torture_kthread_stopping(char *title) >> VERBOSE_TOROUT_STRING(buf); >> while (!kthread_should_stop()) { >> torture_shutdown_absorb(title); >> - schedule_timeout_uninterruptible(1); >> + schedule_timeout_uninterruptible(HZ/20); >> } >> } >> EXPORT_SYMBOL_GPL(torture_kthread_stopping); >> -- >> 2.39.0.314.g84b9a713c41-goog >>
