On 22/04/2021 15:35, Paul E. McKenney wrote:
On Thu, Apr 22, 2021 at 10:20:51AM +0100, John Garry wrote:
Hi RCU experts,
Thanks Paul
Recently I have noticed that I can trigger an RCU stall quite easily on my
system under specific conditions.
I have a fair idea why it happens, but need to analyze a proper solution
further. It looks like a hard IRQ handler and threaded part are tied to
specific CPU and getting swamped and not relinquishing.
However, mixed in the RCU splats, I have noticed many BUG logs, like:
[ 207.788748] BUG: spinlock recursion on CPU#46, fio/1470
This is a self-deadlock. Given that deadlock, and given that spinlocks
disable preemption, the RCU CPU stall warnings are expected behavior.
After all, your code really is grabbing a CPU by the throat and shaking
it indefinitely.
Please build your kernel with CONFIG_PROVE_LOCKING=y and then fix the
issues it calls out. Then please also fix the bugs resulting in the
"sleeping function called from invalid context" and in the "scheduling
while atomic".
Here's the rub, the issue goes away with CONFIG_PROVE_LOCKING and all
the extra debugging it adds. Hmmm.
But I get the point that these are separate and need to be fixed also.
In addition, there are quite a few idle tasks called out in your list of
stalled CPUs. This is often due to RCU's grace-period kthread (named
"rcu_preempt" in this case) not getting any CPU time. This is not
unexpected given the "RT throttling activated". If you are going to run
code at real-time priorities, you must ensure that any number of kernel
kthreads get the CPU time that they need. As Spiderman's uncle said
"With great power comes great responsibility".
OK, I need to check on that separately also.
Cheers,
John
