Patch "rcu: Fix late wakeup when flush of bypass cblist happens" has been added to the 6.1-stable tree

[Sasha Levin <sashal@xxxxxxxxxx>]

This is a note to let you know that I've just added the patch titled

    rcu: Fix late wakeup when flush of bypass cblist happens

to the 6.1-stable tree which can be found at:
    http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
     rcu-fix-late-wakeup-when-flush-of-bypass-cblist-happ.patch
and it can be found in the queue-6.1 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@xxxxxxxxxxxxxxx> know about it.



commit 3f132e8e674299042d9e5313dfbfcb3de55af912
Author: Joel Fernandes (Google) <joel@xxxxxxxxxxxxxxxxx>
Date:   Sat Sep 17 16:41:59 2022 +0000

    rcu: Fix late wakeup when flush of bypass cblist happens
    
    [ Upstream commit b50606f35f4b73c8e4c6b9c64fe7ba72ea919134 ]
    
    When the bypass cblist gets too big or its timeout has occurred, it is
    flushed into the main cblist. However, the bypass timer is still running
    and the behavior is that it would eventually expire and wake the GP
    thread.
    
    Since we are going to use the bypass cblist for lazy CBs, do the wakeup
    soon as the flush for "too big or too long" bypass list happens.
    Otherwise, long delays can happen for callbacks which get promoted from
    lazy to non-lazy.
    
    This is a good thing to do anyway (regardless of future lazy patches),
    since it makes the behavior consistent with behavior of other code paths
    where flushing into the ->cblist makes the GP kthread into a
    non-sleeping state quickly.
    
    [ Frederic Weisbecker: Changes to avoid unnecessary GP-thread wakeups plus
                        comment changes. ]
    
    Reviewed-by: Frederic Weisbecker <frederic@xxxxxxxxxx>
    Signed-off-by: Joel Fernandes (Google) <joel@xxxxxxxxxxxxxxxxx>
    Signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxx>
    Stable-dep-of: cc9b364bb1d5 ("xfrm6: fix inet6_dev refcount underflow problem")
    Signed-off-by: Sasha Levin <sashal@xxxxxxxxxx>

diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 0a5f0ef414845..04c87f250e01a 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 	if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) ||
 	    ncbs >= qhimark) {
 		rcu_nocb_lock(rdp);
+		*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
+
 		if (!rcu_nocb_flush_bypass(rdp, rhp, j)) {
-			*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
 			if (*was_alldone)
 				trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
 						    TPS("FirstQ"));
@@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 			rcu_advance_cbs_nowake(rdp->mynode, rdp);
 			rdp->nocb_gp_adv_time = j;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+
+		// The flush succeeded and we moved CBs into the regular list.
+		// Don't wait for the wake up timer as it may be too far ahead.
+		// Wake up the GP thread now instead, if the cblist was empty.
+		__call_rcu_nocb_wake(rdp, *was_alldone, flags);
+
 		return true; // Callback already enqueued.
 	}