On Mon, Dec 14, 2020 at 01:12:48PM -0500, Joel Fernandes wrote: > On Thu, Dec 10, 2020 at 10:37:37AM -0800, Paul E. McKenney wrote: > > Hello, Joel, > > > > In case you are -seriously- interested... ;-) > > I am always seriously interested :-). The issue becomes when life throws me a > curveball. This was the year of curveballs :-) > > Thank you for your reply and I have added it to my list to investigate how we > are configuring nocb on our systems. I don't think anyone over here has given > these RCU issues a serious look over here. In your defense, I would guess that many of them don't have that much effect. But true, you never know until you try. Thanx, Paul > thanks, > > - Joel > > > > > Thanx, Paul > > > > rcu_nocbs= > > > > Adding a CPU to this list offloads RCU callback invocation from > > that CPU's softirq handler to a kthread. In big.LITTLE systems, > > this kthread can be placed on a LITTLE CPU, which has been > > demonstrated to save significant energy in benchmarks. > > http://www.rdrop.com/users/paulmck/realtime/paper/AMPenergy.2013.04.19a.pdf > > > > nohz_full= > > > > Any CPU specified by this boot parameter is handled as if it was > > specified by rcu_nocbs=. > > > > rcutree.jiffies_till_first_fqs= > > > > Increasing this will decrease wakeup frequency to the grace-period > > kthread for the first FQS scan. And increase grace-period > > latency. > > > > rcutree.jiffies_till_next_fqs= > > > > Ditto, but for the second and subsequent FQS scans. > > > > My guess is that neither of these makes much difference. But if > > they do, maybe some sort of backoff scheme for FQS scans? > > > > rcutree.jiffies_till_sched_qs= > > > > Increasing this will delay RCU's getting excited about CPUs and > > tasks not responding with quiescent states. This excitement > > can cause extra overhead. > > > > No idea whether adjusting this would help. But if you increase > > rcutree.jiffies_till_first_fqs or rcutree.jiffies_till_next_fqs, > > you might need to increase this one accordingly. > > > > rcutree.qovld= > > > > Increasing this will increase the grace-period duration at which > > RCU starts sending IPIs, thus perhaps reducing the total number > > of IPIs that RCU sends. The destination CPUs are unlikely to be > > idle, so it is not clear to me that this would help much. But > > perhaps I am wrong about them being mostly non-idle, who knows? > > > > rcupdate.rcu_cpu_stall_timeout= > > > > If you get overly zealous about the earlier kernel boot parameters, > > you might need to increase this one as well. Or instead use the > > rcupdate.rcu_cpu_stall_suppress= kernel boot parameter to suppress > > RCU CPU stall warnings entirely. > > > > rcutree.rcu_nocb_gp_stride= > > > > Increasing this might reduce grace-period work somewhat. I don't > > see why a (say) 16-CPU system really needs to have more than one > > rcuog kthread, so if this does help it might be worthwhile setting > > a lower limit to this kernel parameter. > > > > rcutree.rcu_idle_gp_delay= (Only CONFIG_RCU_FAST_NO_HZ=y kernels.) > > > > This defaults to four jiffies on the theory that grace periods > > tend to last about that long. If grace periods tend to take > > longer, then it makes a lot of sense to increase this. And maybe > > battery-powered devices would rather have it be about 2x or 3x > > the expected grace-period duration, who knows? > > > > I would keep it to a power of two, but the code should work with > > other numbers. Except that I don't know that this has ever been > > tested. ;-) > > > > srcutree.exp_holdoff= > > > > Increasing this decreases the number of SRCU grace periods that > > are treated as expedited. But you have to have closely-spaced > > SRCU grace periods for this to matter. (These do happen at least > > sometimes because I added this only because someone complained > > about the performance regression from the earlier non-tree SRCU.) > > > > rcupdate.rcu_task_ipi_delay= > > > > This kernel parameter delays sending IPIs for RCU Tasks Trace, > > which is used by sleepable BPF programs. Increasing it can > > reduce overhead, but can also increase the latency of removing > > sleepable BPF programs. > > > > rcupdate.rcu_task_stall_timeout= > > > > If you slow down RCU Tasks Trace too much, you may need this. > > But then again, the default 10-minute value should suffice. > > > > CONFIG_RCU_FAST_NO_HZ=y > > > > This only has effect on CPUs not specified by rcu_nocbs, and thus > > might be useful on systems that offload RCU callbacks only on > > some of the CPUs. For example, a big.LITTLE system might offload > > only the big CPUs. This Kconfig option reduces the frequency of > > timer interrupts (and thus of RCU-related softirq processing) > > on idle CPUs. This has been shown to save significant energy > > in benchmarks: > > http://www.rdrop.com/users/paulmck/realtime/paper/AMPenergy.2013.04.19a.pdf > > > > CONFIG_RCU_STRICT_GRACE_PERIOD=y > > > > This works hard (as in burns CPU) to sharply reduce grace-period > > latency. The effect is probably to greatly increase power > > consumption, but there might well be workloads where the shorter > > grace periods more than make up for the extra CPU time. Or not. > > > > CONFIG_HZ= > > > > Reducing the scheduler-clock interrupt frequency has the opposite > > effect, namely of increasing RCU grace-period latency, but while > > also reducing RCU's CPU utilization. > > > > CONFIG_TASKS_TRACE_RCU_READ_MB=y > > > > Reduce the need to IPI RCU Tasks Trace holdout tasks, but at the > > expense of an increase in to/from idle overhead. This Kconfig > > option also slows down the rate at which RCU Tasks Trace polls > > for holdout tasks. This polling rate cannot be separately > > specified, but if changing the initial source-code values of > > either rcu_tasks_trace.gp_sleep or rcu_tasks_trace.init_fract > > proves useful, kernel boot parameters could be created. > > > > That said, automatic initialization heuristics are more > > convenient. When they work, anyway.
