On Thu, Jun 27, 2019 at 02:16:38PM -0400, Joel Fernandes wrote:
> On Thu, Jun 27, 2019 at 10:38:31AM -0700, Paul E. McKenney wrote:
> > On Thu, Jun 27, 2019 at 12:47:24PM -0400, Joel Fernandes wrote:
> > > On Thu, Jun 27, 2019 at 11:55 AM Paul E. McKenney <paulmck@xxxxxxxxxxxxx> wrote:
> > > >
> > > > On Thu, Jun 27, 2019 at 11:30:31AM -0400, Joel Fernandes wrote:
> > > > > On Thu, Jun 27, 2019 at 10:34:55AM -0400, Steven Rostedt wrote:
> > > > > > On Thu, 27 Jun 2019 10:24:36 -0400
> > > > > > Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote:
> > > > > >
> > > > > > > > What am I missing here?
> > > > > > >
> > > > > > > This issue I think is
> > > > > > >
> > > > > > > (in normal process context)
> > > > > > > spin_lock_irqsave(rq_lock); // which disables both preemption and interrupt
> > > > > > > // but this was done in normal process context,
> > > > > > > // not from IRQ handler
> > > > > > > rcu_read_lock();
> > > > > > > <---------- IPI comes in and sets exp_hint
> > > > > >
> > > > > > How would an IPI come in here with interrupts disabled?
> > > > > >
> > > > > > -- Steve
> > > > >
> > > > > This is true, could it be rcu_read_unlock_special() got called for some
> > > > > *other* reason other than the IPI then?
> > > > >
> > > > > Per Sebastian's stack trace of the recursive lock scenario, it is happening
> > > > > during cpu_acct_charge() which is called with the rq_lock held.
> > > > >
> > > > > The only other reasons I know off to call rcu_read_unlock_special() are if
> > > > > 1. the tick indicated that the CPU has to report a QS
> > > > > 2. an IPI in the middle of the reader section for expedited GPs
> > > > > 3. preemption in the middle of a preemptible RCU reader section
> > > >
> > > > 4. Some previous reader section was IPIed or preempted, but either
> > > > interrupts, softirqs, or preemption was disabled across the
> > > > rcu_read_unlock() of that previous reader section.
> > >
> > > Hi Paul, I did not fully understand 4. The previous RCU reader section
> > > could not have been IPI'ed or been preempted if interrupts were
> > > disabled across. Also, if softirq/preempt is disabled across the
> > > previous reader section, the previous reader could not be preempted in
> > > these case.
> >
> > Like this, courtesy of the consolidation of RCU flavors:
> >
> > previous_reader()
> > {
> > rcu_read_lock();
> > do_something(); /* Preemption happened here. */
> > local_irq_disable(); /* Cannot be the scheduler! */
> > do_something_else();
> > rcu_read_unlock(); /* Must defer QS, task still queued. */
> > do_some_other_thing();
> > local_irq_enable();
> > }
> >
> > current_reader() /* QS from previous_reader() is still deferred. */
> > {
> > local_irq_disable(); /* Might be the scheduler. */
> > do_whatever();
> > rcu_read_lock();
> > do_whatever_else();
> > rcu_read_unlock(); /* Must still defer reporting QS. */
> > do_whatever_comes_to_mind();
> > local_irq_enable();
> > }
> >
> > Both instances of rcu_read_unlock() need to cause some later thing
> > to report the quiescent state, and in some cases it will do a wakeup.
> > Now, previous_reader()'s IRQ disabling cannot be due to scheduler rq/pi
> > locks due to the rule about holding them across the entire RCU reader
> > if they are held across the rcu_read_unlock(). But current_reader()'s
> > IRQ disabling might well be due to the scheduler rq/pi locks, so
> > current_reader() must be careful about doing wakeups.
>
> Makes sense now, thanks.
>
> > > That leaves us with the only scenario where the previous reader was
> > > IPI'ed while softirq/preempt was disabled across it. Is that what you
> > > meant?
> >
> > No, but that can also happen.
> >
> > > But in this scenario, the previous reader should have set
> > > exp_hint to false in the previous reader's rcu_read_unlock_special()
> > > invocation itself. So I would think t->rcu_read_unlock_special should
> > > be 0 during the new reader's invocation thus I did not understand how
> > > rcu_read_unlock_special can be called because of a previous reader.
> >
> > Yes, exp_hint would unconditionally be set to false in the first
> > reader's rcu_read_unlock(). But .blocked won't be.
>
> Makes sense.
>
> > > I'll borrow some of that confused color paint if you don't mind ;-)
> > > And we should document this somewhere for future sanity preservation
> > > :-D
> >
> > Or adjust the code and requirements to make it more sane, if feasible.
> >
> > My current (probably wildly unreliable) guess that the conditions in
> > rcu_read_unlock_special() need adjusting. I was assuming that in_irq()
> > implies a hardirq context, in other words that in_irq() would return
> > false from a threaded interrupt handler. If in_irq() instead returns
> > true from within a threaded interrupt handler, then this code in
> > rcu_read_unlock_special() needs fixing:
> >
> > if ((exp || in_irq()) && irqs_were_disabled && use_softirq &&
> > (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
> > // Using softirq, safe to awaken, and we get
> > // no help from enabling irqs, unlike bh/preempt.
> > raise_softirq_irqoff(RCU_SOFTIRQ);
> >
> > The fix would be replacing the calls to in_irq() with something that
> > returns true only if called from within a hardirq context.
> > Thoughts?
>
> I am not sure if this will fix all cases though?
>
> I think the crux of the problem is doing a recursive wake up. The threaded
> IRQ probably just happens to be causing it here, it seems to me this problem
> can also occur on a non-threaded irq system (say current_reader() in your
> example executed in a scheduler path in process-context and not from an
> interrupt). Is that not possible?
In the non-threaded case, invoking raise_softirq*() from hardirq context
just sets a bit in a per-CPU variable. Now, to Sebastian's point, we
are only sort of in hardirq context in this case due to being called
from irq_exit(), but the failure we are seeing might well be a ways
downstream of the actual root-cause bug.
> I think the fix should be to prevent the wake-up not based on whether we are
> in hard/soft-interrupt mode but that we are doing the rcu_read_unlock() from
> a scheduler path (if we can detect that)
Or just don't do the wakeup at all, if it comes to that. I don't know
of any way to determine whether rcu_read_unlock() is being called from
the scheduler, but it has been some time since I asked Peter Zijlstra
about that.
Of course, unconditionally refusing to do the wakeup might not be happy
thing for NO_HZ_FULL kernels that don't implement IRQ work.
> I lost track of this code:
> if ((exp || in_irq()) && irqs_were_disabled && use_softirq &&
> (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
>
> Was this patch posted to the list? I will blame it to try to get some
> context. It sounds like you added more conditions on when to kick the
> softirq.
This is from the dev branch of my -rcu tree. It has at least one
patch in this area that is currently slated for v5.4, so I would not
have sent that as part of an official patch series.
> > Ugh. Same question about IRQ work. Will the current use of it by
> > rcu_read_unlock_special() cause breakage in the presence of threaded
> > interrupt handlers?
>
> /me needs to understand why the irq work stuff was added here as well. Have
> my work cut out for the day! ;-)
New code, so more likely to contain bugs than usual. ;-)
The point was to get a wakeup soonish without risk of rq/pi deadlocks.
Thanx, Paul
> thanks,
>
> - Joel
>
>
> >
> > Thanx, Paul
> >
> > > thanks,
> > > - Joel
> > >
> > >
> > >
> > > >
> > > > I -think- that this is what Sebastian is seeing.
> > > >
> > > > Thanx, Paul
> > > >
> > > > > 1. and 2. are not possible because interrupts are disabled, that's why the
> > > > > wakeup_softirq even happened.
> > > > > 3. is not possible because we are holding rq_lock in the RCU reader section.
> > > > >
> > > > > So I am at a bit of a loss how this can happen :-(
> > > > >
> > > > > Spurious call to rcu_read_unlock_special() may be when it should not have
> > > > > been called?
> > > > >
> > > > > thanks,
> > > > >
> > > > > - Joel
> >
>
