On Sun, Aug 04, 2019 at 01:24:46PM -0700, Paul E. McKenney wrote: > On Sun, Aug 04, 2019 at 11:41:59AM -0700, Paul E. McKenney wrote: > > On Sun, Aug 04, 2019 at 04:48:35PM +0200, Peter Zijlstra wrote: > > > On Sun, Aug 04, 2019 at 04:43:17PM +0200, Peter Zijlstra wrote: > > > > On Fri, Aug 02, 2019 at 08:15:01AM -0700, Paul E. McKenney wrote: > > > > > The multi_cpu_stop() function relies on the scheduler to gain control from > > > > > whatever is running on the various online CPUs, including any nohz_full > > > > > CPUs running long loops in kernel-mode code. Lack of the scheduler-clock > > > > > interrupt on such CPUs can delay multi_cpu_stop() for several minutes > > > > > and can also result in RCU CPU stall warnings. This commit therefore > > > > > causes multi_cpu_stop() to enable the scheduler-clock interrupt on all > > > > > online CPUs. > > > > > > > > This sounds wrong; should we be fixing sched_can_stop_tick() instead to > > > > return false when the stop task is runnable? > > > > Agreed. However, it is proving surprisingly hard to come up with a > > code sequence that has the effect of rcu_nocb without nohz_full. > > And rcu_nocb works just fine. With nohz_full also in place, I am > > decreasing the failure rate, but it still fails, perhaps a few times > > per hour of TREE04 rcutorture on an eight-CPU system. (My 12-CPU > > system stubbornly refuses to fail. Good thing I kept the eight-CPU > > system around, I guess.) > > > > When I arrive at some sequence of actions that actually work reliably, > > then by all means let's put it somewhere in the NO_HZ_FULL machinery! > > > > > And even without that; I don't understand how we're not instantly > > > preempted the moment we enqueue the stop task. > > > > There is no preemption because CONFIG_PREEMPT=n for the scenarios still > > having trouble. Yes, there are cond_resched() calls, but they don't do > > anything unless the appropriate flags are set, which won't always happen > > without the tick, apparently. Or without -something- that isn't always > > happening as it should. > > > > > Any enqueue, should go through check_preempt_curr() which will be an > > > instant resched_curr() when we just woke the stop class. > > > > I did try hitting all of the CPUs with resched_cpu(). Ten times on each > > CPU with a ten-jiffy wait between each. This might have decreased the > > probability of excessively long CPU-stopper waits by a factor of two or > > three, but it did not eliminate the excessively long waits. > > > > What else should I try? > > > > For example, are there any diagnostics I could collect, say from within > > the CPU stopper when things are taking too long? I see CPU-stopper > > delays in excess of five -minutes-, so this is anything but subtle. > > For whatever it is worth, the things on my list include using 25 rounds > of resched_cpu() on each CPU with ten-jiffy wait between each (instead of > merely 10 rounds), using waitqueues or some such to actually force a > meaningful context switch on the other CPUs, etc. Which appears to have reduced the bug rate by about a factor of two. (But statistics and all that.) I am now trying the same test, but with CONFIG_PREEMPT=y and without quite so much hammering on the scheduler. This is keying off Peter's earlier mention of preemption. If this turns out to be solid, perhaps we outlaw CONFIG_PREEMPT=n && CONFIG_NO_HZ_FULL=y? Thanx, Paul
