On Thu, Oct 06, 2022 at 01:31:43PM +0000, Joel Fernandes wrote: > On Wed, Oct 05, 2022 at 06:03:39PM +0200, Daniel Bristot de Oliveira wrote: > > On 10/5/22 18:01, Florent Carli wrote: > > > Hello, > > > > > > I'm trying to isolate some cores to run a CPU-bound real-time task and > > > even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc > > > threads wake up, and I get RCU stall warnings on the isolated core. > > Isolating a CPU does not mean that RCU activity is forbidden on it, even with > nohz_full, if the CPU enters the kernel (say syscall or interrupt), then it > may enter a read-side critical section so RCU will be watching it, and such > CPU will have to report quiescent state. What Joel said! Plus, if you keep that CPU out of the kernel (including avoiding it ever taking any interrupts), you won't see rcuc activity on that CPU. > > > As per the documentation, if I use rcutree.kthread_prio with a > > > priority > my RT task, then the rcu stall does not happen. > > Yes, it sounds like the main RCU GP thread (mostly called rcu_preempt thread) > in your system is competing with your RT task. Increase kthread_prio is > standard procedure to resolve this issue, as you did. Or affinity the rcu_preempt kthread elsewhere so that it is not competing with your real-time workload. > > > However I find it confusing: why are the rcuc threads woken up on the > > > isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very > > > likely erroneous) understanding, I shouldn't have to fiddle with rcu > > > priority... In other words, how come I get rcu stalls on cores that > > > have no rcu callbacks? > > It is not possible to provide further help without more info, in particular > are you using nohz_full and isolcpus options? Can you provide kernel > and #CPU configuration? > > Happy to look further! Also adding some more folks who know a lot about this > stuff and +rcu list for archives. The rcuc kthreads communicate between the corresponding CPU and the rcu_preempt kthread (AKA RCU GP kthread). They wake up only when there is something for them to do, and letting your nohz_full/rcu_nocbs CPUs execute in the kernel is an excellent way to give them something to do. In particular, if that CPU runs in the kernel, it might have executed an RCU read-side critical section. Because RCU grace periods are not permitted to complete until all pre-existing RCU read-side critical sections finish, communication is required. So, yes, that rcuc kthread will need to execute if its CPU queues an RCU callback. But it might also need to execute if some other CPU queues an RCU callback -and- its CPU spends significant time in the kernel. Thanx, Paul > thanks, > > - Joel > > > > > Adding Joel because we had a chat about it during lpc... > > > > > Thanks. > > > > > -- Daniel > >
