The pair of segcblist operations: rcu_segcblist_advance() and
rcu_segcblist_accelerate() in srcu_gp_start() is needless from two
perspectives:
-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed() has called that pair operations and attached
it with gp_seq. At the exit, srcu_invoke_callbacks() calls that pair
again to extract the done callbacks. So the harvesting of the callback
is not affected by the call to that pair in srcu_gp_start().
But because the updating of RCU_DONE_TAIL by srcu_invoke_callbacks() may
have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may
cause srcu_might_be_idle() not to be real time. To compensate that,
supplement that pair just before the calling to rcu_segcblist_pend_cbs()
in srcu_might_be_idle().
Test info:
torture test passed using the following command against commit 094226ad94f4 ("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 10h --configs 18*SRCU-P
Signed-off-by: Pingfan Liu <kernelfans@xxxxxxxxx>
Cc: Lai Jiangshan <jiangshanlai@xxxxxxxxx>
Cc: "Paul E. McKenney" <paulmck@xxxxxxxxxx>
Cc: Frederic Weisbecker <frederic@xxxxxxxxxx>
Cc: Josh Triplett <josh@xxxxxxxxxxxxxxxx>
Cc: Steven Rostedt <rostedt@xxxxxxxxxxx>
Cc: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
To: rcu@xxxxxxxxxxxxxxx
---
kernel/rcu/srcutree.c | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 725c82bb0a6a..36ba18967133 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);
*/
static void srcu_gp_start(struct srcu_struct *ssp)
{
- struct srcu_data *sdp;
int state;
- if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER)
- sdp = per_cpu_ptr(ssp->sda, 0);
- else
- sdp = this_cpu_ptr(ssp->sda);
lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock));
WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed));
- spin_lock_rcu_node(sdp); /* Interrupts already disabled. */
- rcu_segcblist_advance(&sdp->srcu_cblist,
- rcu_seq_current(&ssp->srcu_gp_seq));
- (void)rcu_segcblist_accelerate(&sdp->srcu_cblist,
- rcu_seq_snap(&ssp->srcu_gp_seq));
- spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */
WRITE_ONCE(ssp->srcu_gp_start, jiffies);
WRITE_ONCE(ssp->srcu_n_exp_nodelay, 0);
smp_mb(); /* Order prior store to ->srcu_gp_seq_needed vs. GP start. */
@@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)
/* If the local srcu_data structure has callbacks, not idle. */
sdp = raw_cpu_ptr(ssp->sda);
spin_lock_irqsave_rcu_node(sdp, flags);
+ rcu_segcblist_advance(&sdp->srcu_cblist,
+ rcu_seq_current(&ssp->srcu_gp_seq));
+ (void)rcu_segcblist_accelerate(&sdp->srcu_cblist,
+ rcu_seq_snap(&ssp->srcu_gp_seq));
if (rcu_segcblist_pend_cbs(&sdp->srcu_cblist)) {
spin_unlock_irqrestore_rcu_node(sdp, flags);
return false; /* Callbacks already present, so not idle. */
--
2.31.1
