On Fri, Jul 05, 2024 at 06:12:40PM +0200, Sebastian Andrzej Siewior wrote:
> Hi,
>
> I noticed this on an older kernel and reproduced it on -rc6: I boot the
> (RT) kernel with
> isolcpus=4-15 nohz=on nohz_full=4-15 rcu_nocbs=4-15 rcu_nocb_poll
>
> and compiled a NOHZ_FULL kernel. If I pin a shell onto the isolated CPU
> and do nothing then everything seems fine. If I issue commands, this
> happens:
> tick_nohz_handler() -> update_process_times() -> rcu_sched_clock_irq()
>
> rcu_sched_clock_irq() invokes rcu_pending() which returns true due to
> | /* Have RCU grace period completed or started? */
> | if (rcu_seq_current(&rnp->gp_seq) != rdp->gp_seq ||
> | unlikely(READ_ONCE(rdp->gpwrap))) /* outside lock */
> | return 1;
>
> And it is the first check of the two. Later I see
> | /* Is the RCU core waiting for a quiescent state from this CPU? */
> | gp_in_progress = rcu_gp_in_progress();
> | if (rdp->core_needs_qs && !rdp->cpu_no_qs.b.norm && gp_in_progress)
> | return 1;
>
> which triggers it but then "later" it fallbacks to the first case again.
> In both cases it wakes the rcuc/ thread on the isolated CPU.
>
> Is this something that needs to happen on the isolated CPU or can this
> be excluded for rcu_rdp_is_offloaded() CPUs?
As a workaround, the following commit in -rcu that is slated for
the upcoming merge window addresses a similar case involving KVM and
nohz_full:
68d124b09999 ("rcu: Add rcutree.nohz_full_patience_delay to reduce
nohz_full OS jitter")
The KVM guys found that setting rcutree.nohz_full_patience_delay to 1000
(AKA one second) made things work better for them. Does this help your
use case?
This is again a workaround. Clearly, it would be better if we could
eliminate that second rcuc wakeup. I tried something similar some time
back, and there was a problem with it. I will see if I can reconstitute
the corresponding brain cells.
But in the meantime, one advantage of the workaround is that in the
common case, it would reduce the number of rcuc wakeups to zero, rather
than to just one.
Thoughts?
Thanx, Paul
