RCU experts, When you reply, please also keep me CC'ed. The problem of RCU stall might be an old problem and it can happen quite often. As I have observed, when the problem occurs, at least one CPU in the system on which its rdp->gp_seq falls behind others by 4 (qs). e.g. On CPU 0, rdp->gp_seq = 0x13889d, but on other CPUs, their rdp->gp_seq = 0x1388a1. Because RCU stall issues can last a long period of time, the number of callbacks in the list rdp->cblist of all CPUs can accumulate to thousands. In the worst case, it triggers panic. When looking into the problem further, I'd think the problem is related to the Linux scheduler. When the RCU core detects the stall on a CPU, rcu_gp_kthread would send a rescheduling request via send_IPI to that CPU to try to force a context switch to make some progress. However, at least one situation can fail this effort, which is when the CPU is running a user thread and it is the only user thread in the rq, then this attempted context switching will not happen immediately. In particular if the system is also configured with NOHZ_FULL for the CPU and as long as the user thread is running, the forced context switch will never happen unless the user thread volunteers to yield the CPU. I think this should be one of the major root causes of these RCU stall issues. Even if NOHZ_FULL is not configured, there will be at least 1 tick delay which can affect the realtime kernel, by the way. But it seems not a good idea to craft a fix from the scheduler side because this has to invalidate some existing scheduling optimizations. The current scheduler is deliberately optimized to avoid such context switching. So my question is why the RCU core cannot effectively update qs for the stalled CPU when it detects that the stalled CPU is running a user thread? The reason is pretty obvious because when a CPU is running a user thread, it must not be in any kernel read-side critical sections. So it should be safe to close its current RCU grace period on this CPU. Also, with this approach we can make RCU work more efficiently than the approach of context switch which needs to go through an IPI interrupt and the destination CPU needs to wake up its ksoftirqd or wait for the next scheduling cycle. If my suggested approach makes sense, I can go ahead to fix it that way. Thanks Donghai
