Thank you all. I make sure the rcu_preempt kthread affinity was set on a non-isolated core (as well as all unbounded rcu* threads) so I guess it is not a competing issue. What I’m running on those isolated cores are qemu/kvm vcpu tasks, and I guess they might spend some time in the kernel… in which case I understand it’s completely normal to see some rcuc activity. And so I conclude that using isolcpus + nohz_full (which includes rcu_nocbs) + rcu_nocb_poll + rcutree.kthread_prio is the way to go. Thanks everyone. On Thu, Oct 6, 2022 at 4:13 PM Paul E. McKenney <paulmck@xxxxxxxxxx> wrote: > > On Thu, Oct 06, 2022 at 01:31:43PM +0000, Joel Fernandes wrote: > > On Wed, Oct 05, 2022 at 06:03:39PM +0200, Daniel Bristot de Oliveira wrote: > > > On 10/5/22 18:01, Florent Carli wrote: > > > > Hello, > > > > > > > > I'm trying to isolate some cores to run a CPU-bound real-time task and > > > > even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc > > > > threads wake up, and I get RCU stall warnings on the isolated core. > > > > Isolating a CPU does not mean that RCU activity is forbidden on it, even with > > nohz_full, if the CPU enters the kernel (say syscall or interrupt), then it > > may enter a read-side critical section so RCU will be watching it, and such > > CPU will have to report quiescent state. > > What Joel said! > > Plus, if you keep that CPU out of the kernel (including avoiding it ever > taking any interrupts), you won't see rcuc activity on that CPU. > > > > > As per the documentation, if I use rcutree.kthread_prio with a > > > > priority > my RT task, then the rcu stall does not happen. > > > > Yes, it sounds like the main RCU GP thread (mostly called rcu_preempt thread) > > in your system is competing with your RT task. Increase kthread_prio is > > standard procedure to resolve this issue, as you did. > > Or affinity the rcu_preempt kthread elsewhere so that it is not competing > with your real-time workload. > > > > > However I find it confusing: why are the rcuc threads woken up on the > > > > isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very > > > > likely erroneous) understanding, I shouldn't have to fiddle with rcu > > > > priority... In other words, how come I get rcu stalls on cores that > > > > have no rcu callbacks? > > > > It is not possible to provide further help without more info, in particular > > are you using nohz_full and isolcpus options? Can you provide kernel > > and #CPU configuration? > > > > Happy to look further! Also adding some more folks who know a lot about this > > stuff and +rcu list for archives. > > The rcuc kthreads communicate between the corresponding CPU and the > rcu_preempt kthread (AKA RCU GP kthread). They wake up only when there > is something for them to do, and letting your nohz_full/rcu_nocbs CPUs > execute in the kernel is an excellent way to give them something to do. > > In particular, if that CPU runs in the kernel, it might have executed > an RCU read-side critical section. Because RCU grace periods are not > permitted to complete until all pre-existing RCU read-side critical > sections finish, communication is required. > > So, yes, that rcuc kthread will need to execute if its CPU queues an > RCU callback. But it might also need to execute if some other CPU queues > an RCU callback -and- its CPU spends significant time in the kernel. > > Thanx, Paul > > > thanks, > > > > - Joel > > > > > > > > Adding Joel because we had a chat about it during lpc... > > > > > > > Thanks. > > > > > > > -- Daniel > > >
