On Sun, Aug 04, 2019 at 11:41:59AM -0700, Paul E. McKenney wrote: > On Sun, Aug 04, 2019 at 04:48:35PM +0200, Peter Zijlstra wrote: > > On Sun, Aug 04, 2019 at 04:43:17PM +0200, Peter Zijlstra wrote: > > > On Fri, Aug 02, 2019 at 08:15:01AM -0700, Paul E. McKenney wrote: > > > > The multi_cpu_stop() function relies on the scheduler to gain control from > > > > whatever is running on the various online CPUs, including any nohz_full > > > > CPUs running long loops in kernel-mode code. Lack of the scheduler-clock > > > > interrupt on such CPUs can delay multi_cpu_stop() for several minutes > > > > and can also result in RCU CPU stall warnings. This commit therefore > > > > causes multi_cpu_stop() to enable the scheduler-clock interrupt on all > > > > online CPUs. > > > > > > This sounds wrong; should we be fixing sched_can_stop_tick() instead to > > > return false when the stop task is runnable? > > Agreed. However, it is proving surprisingly hard to come up with a > code sequence that has the effect of rcu_nocb without nohz_full. > And rcu_nocb works just fine. With nohz_full also in place, I am > decreasing the failure rate, but it still fails, perhaps a few times > per hour of TREE04 rcutorture on an eight-CPU system. (My 12-CPU > system stubbornly refuses to fail. Good thing I kept the eight-CPU > system around, I guess.) > > When I arrive at some sequence of actions that actually work reliably, > then by all means let's put it somewhere in the NO_HZ_FULL machinery! > > > And even without that; I don't understand how we're not instantly > > preempted the moment we enqueue the stop task. > > There is no preemption because CONFIG_PREEMPT=n for the scenarios still > having trouble. Yes, there are cond_resched() calls, but they don't do > anything unless the appropriate flags are set, which won't always happen > without the tick, apparently. Or without -something- that isn't always > happening as it should. > > > Any enqueue, should go through check_preempt_curr() which will be an > > instant resched_curr() when we just woke the stop class. > > I did try hitting all of the CPUs with resched_cpu(). Ten times on each > CPU with a ten-jiffy wait between each. This might have decreased the > probability of excessively long CPU-stopper waits by a factor of two or > three, but it did not eliminate the excessively long waits. > > What else should I try? > > For example, are there any diagnostics I could collect, say from within > the CPU stopper when things are taking too long? I see CPU-stopper > delays in excess of five -minutes-, so this is anything but subtle. For whatever it is worth, the things on my list include using 25 rounds of resched_cpu() on each CPU with ten-jiffy wait between each (instead of merely 10 rounds), using waitqueues or some such to actually force a meaningful context switch on the other CPUs, etc. Thanx, Paul
