On 2025-02-04 03:51:48 [-0800], Paul E. McKenney wrote:
> On Tue, Feb 04, 2025 at 11:26:11AM +0100, Sebastian Andrzej Siewior wrote:
> > On 2025-01-30 10:53:19 [-0800], Paul E. McKenney wrote:
> > > The timer and hrtimer softirq processing has moved to dedicated threads
> > > for kernels built with CONFIG_IRQ_FORCED_THREADING=y. This results in
> > > timers not expiring until later in early boot, which in turn causes the
> > > RCU Tasks self-tests to hang in kernels built with CONFIG_PROVE_RCU=y,
> > > which further causes the entire kernel to hang. One fix would be to
> > > make timers work during this time, but there are no known users of RCU
> > > Tasks grace periods during that time, so no justification for the added
> > > complexity. Not yet, anyway.
> > >
> > > This commit therefore moves the call to rcu_init_tasks_generic() from
> > > kernel_init_freeable() to a core_initcall(). This works because the
> > > timer and hrtimer kthreads are created at early_initcall() time.
> >
> > Fixes: 49a17639508c3 ("softirq: Use a dedicated thread for timer wakeups on PREEMPT_RT.")
> > ?
>
> Quite possibly... I freely confess that I was more focused on the fix
> than on the bug's origin. Would you be willing to try this commit and
> its predecessor?
Yes. Just verified.
Tested-by: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
> > I played with it and I can reproduce the issue with !RT + threadirqs but
> > not with RT (which implies threadirqs).
> > Is there anything in RT that avoids the problem?
>
> Not that I know of, but then again I did not try it. To your point,
The change looks fine.
> I do need to make a -rt rcutorture scenario. TREE03 has been intended to
> approximate this, and it uses the following Kconfig options:
>
> ------------------------------------------------------------------------
>
> CONFIG_SMP=y
> CONFIG_NR_CPUS=16
> CONFIG_PREEMPT_NONE=n
> CONFIG_PREEMPT_VOLUNTARY=n
> CONFIG_PREEMPT=y
> #CHECK#CONFIG_PREEMPT_RCU=y
> CONFIG_HZ_PERIODIC=y
> CONFIG_NO_HZ_IDLE=n
> CONFIG_NO_HZ_FULL=n
> CONFIG_RCU_TRACE=y
> CONFIG_HOTPLUG_CPU=y
> CONFIG_RCU_FANOUT=2
> CONFIG_RCU_FANOUT_LEAF=2
> CONFIG_RCU_NOCB_CPU=n
> CONFIG_DEBUG_LOCK_ALLOC=n
> CONFIG_RCU_BOOST=y
> CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
> CONFIG_RCU_EXPERT=y
You could enable CONFIG_PREEMPT_RT ;)
CONFIG_PREEMPT_LAZY is probably also set a lot.
That should be it.
> ------------------------------------------------------------------------
>
> And the following kernel-boot parameters:
>
> ------------------------------------------------------------------------
>
> rcutorture.onoff_interval=200 rcutorture.onoff_holdoff=30
> rcutree.gp_preinit_delay=12
> rcutree.gp_init_delay=3
> rcutree.gp_cleanup_delay=3
> rcutree.kthread_prio=2
> threadirqs
> rcutree.use_softirq=0
> rcutorture.preempt_duration=10
>
> ------------------------------------------------------------------------
>
> Some of these are for RCU's benefit, but what should I change to more
> closely approximate a typical real-time deployment?
See above.
> > Thank you for debugging and the patch.
>
> And to you for digging into it!
Thanks.
> Thanx, Paul
Sebastian
