On Mon, Feb 22, 2021 at 06:16:05PM +0100, Uladzislau Rezki wrote:
> On Mon, Feb 22, 2021 at 07:09:03AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 22, 2021 at 01:54:31PM +0100, Uladzislau Rezki wrote:
> > > On Mon, Feb 22, 2021 at 11:21:04AM +0100, Sebastian Andrzej Siewior wrote:
> > > > On 2021-02-19 10:33:36 [-0800], Paul E. McKenney wrote:
> > > > > For definiteness, here is the first part of the change, posted earlier.
> > > > > The commit log needs to be updated. I will post the change that keeps
> > > > > the tick going as a reply to this email.
> > > > …
> > > > > diff --git a/kernel/softirq.c b/kernel/softirq.c
> > > > > index 9d71046..ba78e63 100644
> > > > > --- a/kernel/softirq.c
> > > > > +++ b/kernel/softirq.c
> > > > > @@ -209,7 +209,7 @@ static inline void invoke_softirq(void)
> > > > > if (ksoftirqd_running(local_softirq_pending()))
> > > > > return;
> > > > >
> > > > > - if (!force_irqthreads) {
> > > > > + if (!force_irqthreads || !__this_cpu_read(ksoftirqd)) {
> > > > > #ifdef CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
> > > > > /*
> > > > > * We can safely execute softirq on the current stack if
> > > > > @@ -358,8 +358,8 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
> > > > >
> > > > > pending = local_softirq_pending();
> > > > > if (pending) {
> > > > > - if (time_before(jiffies, end) && !need_resched() &&
> > > > > - --max_restart)
> > > > > + if (!__this_cpu_read(ksoftirqd) ||
> > > > > + (time_before(jiffies, end) && !need_resched() && --max_restart))
> > > > > goto restart;
> > > >
> > > > This is hunk shouldn't be needed. The reason for it is probably that the
> > > > following wakeup_softirqd() would avoid further invoke_softirq()
> > > > performing the actual softirq work. It would leave early due to
> > > > ksoftirqd_running(). Unless I'm wrong, any raise_softirq() invocation
> > > > outside of an interrupt would do the same.
> >
> > And it does pass the rcutorture test without that hunk:
> >
> > tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 2 --configs "TREE03" --kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y" --bootargs "threadirqs=1" --trust-make
> >
> Yep. I have tested that patch also. It works for me as well. So
> technically i do not see any issues from the first glance but of
> course it should be reviewed by the softirq people to hear their
> opinion.
>
> IRQs are enabled, so it can be handled from an IRQ tail until
> ksoftirqd threads are spawned.
And if I add "CONFIG_NO_HZ_IDLE=y CONFIG_HZ_PERIODIC=n" it still works,
even if I revert my changes to rcu_needs_cpu(). Should I rely on this
working globally? ;-)
Thanx, Paul
> > > > I would like PeterZ / tglx to comment on this one. Basically I'm not
> > > > sure if it is okay to expect softirqs beeing served and waited on that
> > > > early in the boot.
> >
> > It would be good to get other eyes on this.
> >
> > I do agree that "don't wait on softirq handlers until after completion
> > of all early_initcall() handlers" is a nice simple rule, but debugging
> > violations of it is not so simple. Adding warnings to ease debugging
> > of violations of this rule is quite a bit more complex than is either of
> > the methods of making the rule unnecessary, at least from what I can see
> > at this point. The complexity of the warnings is exactly what Sebastian
> > pointed out earlier, that it is currently legal to raise_softirq() but
> > not to wait on the resulting handlers. But even waiting is OK if that
> > waiting does not delay the boot sequence. But if the boot kthread waits
> > on the kthread that does the waiting, it is once again not OK.
> >
> > So am I missing something subtle here?
> >
> I agree here. Seems like we are on the same page in understanding :)
>
> > > The ksoftirqd threads get spawned during early_initcall() phase. Why not
> > > just spawn them one step earlier what is totally safe? I mean before
> > > do_pre_smp_initcalls() that calls early callbacks.
> > >
> > > + spawn_ksoftirqd();
> > > rcu_init_tasks_generic();
> > > do_pre_smp_initcalls();
> > >
> > > With such change the spawning will not be depended on linker/compiler
> > > i.e. when and in which order an early_initcall(spawn_ksoftirqd) callback
> > > is executed.
> >
> > We both posted patches similar to this, so I am not opposed. One caveat,
> > though, namely that this narrows the window quite a bit but does not
> > entirely close it. But it does allow the early_initcall()s to wait on
> > softirq handlers.
> >
> Yep, that was an intention. At least to provide such functionality for early
> callbacks. What happens before it(init/main.c) is pretty controllable.
>
> --
> Vlad Rezki
