Thanks Paul for the insights!
I studied the 3 options and think that #1+#3 offers both flexibility
to users and coverage of boundary user cases.
For example, as an user of RCU, we want the warnings to be spilled at
the default 21 seconds so that we know such events are happening. At
the same time, we want Panic to happen if the stall is long enough to
significantly affect available system memory on our system.
Here is the plan based on our discussion, please advise if not inline
with the idea:
1. modify panic_on_rcu_stall to be the maximum number of consecutive
warnings to trigger Panic.
1) change its name to max_rcu_stall_to_panic,
2) default value to 1, which is the same behavior as today's.
2. use ((struct rcu_state *)->gpnum - (struct rcu_data *)->gpnum) >=
max_rcu_stall_to_panic as condition to trigger Panic;
3. reset (struct rcu_data *)->gpnum to (struct rcu_state *)->gpnum
every time a new grace period starts;
4. add a new member (struct rcu_data *)->gpmiss that is incremented at
each miss to track how many misses so far for statistics/debug
purpose.
Your insights and advice are highly appreciated.
Thanks!
Chao
On Thu, Aug 13, 2020 at 11:19 AM Paul E. McKenney <paulmck@xxxxxxxxxx> wrote:
>
> On Thu, Aug 13, 2020 at 10:22:09AM -0700, Chao Zhou wrote:
> > Hi,
> >
> > Some RCU stalls are transient and a system is fully capable to recover
> > after that, but we do want Panic after certain amount of GP misses.
> >
> > Current module parameter rcu_cpu_stall_panic only turn on/off Panic,
> > and 1 GP miss will trigger Panic when it is enabled.
> >
> > Plan to add a module parameter for users to fine-tune how many GP
> > misses are allowed before Panic.
> >
> > To save our precious time, a diff has been tested on our systems and
> > it works and solves our problem in transient RCU stall events.
> >
> > Your insights and guidance is highly appreciated.
>
> Please feel free to post a patch. I could imagine a number of things
> you might be doing from your description above:
>
> 1. Having a different time for panic, so that (for example) an
> RCU CPU stall warning appears at 21 seconds (in mainline), and
> if the grace period still has not ended at some time specified
> by some kernel parameter. For example, one approach would be
> to make the existing panic_on_rcu_stall sysctl take an integer
> instead of a boolean, and to make that integer specify how old
> the stall-warned grace period must be before panic() is invoked.
>
> 2. Instead use the number of RCU CPU stall warning messages to
> trigger the panic, so that (for example), the panic would happen
> on the tenth message. Again, the panic_on_rcu_stall sysctl
> might be used for this.
>
> 3. Like #2, but reset the count every time a new grace period
> starts. So if the panic_on_rcu_stall sysctl was set to
> ten, there would need to be ten RCU CPU stall warnings for
> the same grace period before panic() was invoked.
>
> Of the above three, #1 and #3 seem the most attractive, with a slight
> preference for #1.
>
> Or did you have something else in mind?
>
> Thanx, Paul
