On Mon, Aug 14, 2023 at 03:39:31AM +0000, Joel Fernandes (Google) wrote:
> From: Joel Fernandes <joel@xxxxxxxxxxxxxxxxx>
>
> [ Upstream commit d52d3a2bf408ff86f3a79560b5cce80efb340239 ]
> During shutdown of rcutorture, the shutdown thread in
> rcu_torture_cleanup() calls torture_cleanup_begin() which sets fullstop
> to FULLSTOP_RMMOD. This is enough to cause the rcutorture threads for
> readers and fakewriters to breakout of their main while loop and start
> shutting down.
>
> Once out of their main loop, they then call torture_kthread_stopping()
> which in turn waits for kthread_stop() to be called, however
> rcu_torture_cleanup() has not even called kthread_stop() on those
> threads yet, it does that a bit later. However, before it gets a chance
> to do so, torture_kthread_stopping() calls
> schedule_timeout_interruptible(1) in a tight loop. Tracing confirmed
> this makes the timer softirq constantly execute timer callbacks, while
> never returning back to the softirq exit path and is essentially "locked
> up" because of that. If the softirq preempts the shutdown thread,
> kthread_stop() may never be called.
>
> This commit improves the situation dramatically, by increasing timeout
> passed to schedule_timeout_interruptible() 1/20th of a second. This
> causes the timer softirq to not lock up a CPU and everything works fine.
> Testing has shown 100 runs of TREE07 passing reliably, which was not the
> case before because of RCU stalls.
>
> Cc: Paul McKenney <paulmck@xxxxxxxxxx>
> Cc: Frederic Weisbecker <fweisbec@xxxxxxxxx>
> Cc: Zhouyi Zhou <zhouzhouyi@xxxxxxxxx>
> Cc: <stable@xxxxxxxxxxxxxxx> # 6.0.x
> Signed-off-by: Joel Fernandes (Google) <joel@xxxxxxxxxxxxxxxxx>
> Reviewed-by: Davidlohr Bueso <dave@xxxxxxxxxxxx>
> Tested-by: Zhouyi Zhou <zhouzhouyi@xxxxxxxxx>
> ---
> kernel/torture.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/torture.c b/kernel/torture.c
> index 1061492f14bd..477d9b601438 100644
> --- a/kernel/torture.c
> +++ b/kernel/torture.c
> @@ -788,7 +788,7 @@ void torture_kthread_stopping(char *title)
> VERBOSE_TOROUT_STRING(buf);
> while (!kthread_should_stop()) {
> torture_shutdown_absorb(title);
> - schedule_timeout_uninterruptible(1);
> + schedule_timeout_uninterruptible(HZ/20);
> }
> }
> EXPORT_SYMBOL_GPL(torture_kthread_stopping);
> --
> 2.41.0.640.ga95def55d0-goog
>
All now queued up, thanks.
greg k-h
