On Wed, Feb 17, 2021 at 11:47:59PM +0900, Masami Hiramatsu wrote:
> On Tue, 16 Feb 2021 09:30:03 -0800
> "Paul E. McKenney" <paulmck@xxxxxxxxxx> wrote:
>
> > On Mon, Feb 15, 2021 at 12:28:26PM +0100, Sebastian Andrzej Siewior wrote:
> > > On 2021-02-13 08:45:54 [-0800], Paul E. McKenney wrote:
> > > > Glad you like it! But let's see which (if any) of these patches solves
> > > > the problem for Sebastian.
> > >
> > > Looking at that, is there any reason for doing this that can not be
> > > solved by moving the self-test a little later? Maybe once we reached at
> > > least SYSTEM_SCHEDULING?
> >
> > One problem is that ksoftirqd and the kprobes use are early_initcall(),
> > so we cannot count on ksoftirqd being spawned when kprobes first uses
> > synchronize_rcu_tasks(). Moving the selftest later won't fix this
> > problem, but rather just paper it over.
> >
> > > This happens now even before lockdep is up or the console is registered.
> > > So if something bad happens, you end up with a blank terminal.
> >
> > I was getting a splat, but I could easily believe that there are
> > configurations where the hang is totally silent. In other words, I do
> > agree that this needs a proper fix. All we need do is work out an
> > agreeable value of "proper". ;-)
> >
> > > There is nothing else that early in the boot process that requires
> > > working softirq. The only exception to this is wait_task_inactive()
> > > which is used while starting a new thread (including the ksoftirqd)
> > > which is why it was moved to schedule_hrtimeout().
> >
> > Moving kprobes initialization to early_initcall() [1] means that there
> > can be a call to synchronize_rcu_tasks() before the current spawning of
> > ksoftirqd. Because synchronize_rcu_tasks() needs timers to work, it needs
> > softirq to work. I know two straightforward ways to make that happen:
> >
> > 1. Spawn ksoftirqd earlier.
> >
> > 2. Suppress attempts to awaken ksoftirqd before it exists,
> > forcing all ksoftirq execution on the back of interrupts.
> >
> > Uladzislau and I each produced patches for #1, and I produced a patch
> > for #2.
> >
> > The only other option I know of is to push the call to init_kprobes()
> > later in the boot sequence, perhaps to its original subsys_initcall(),
> > or maybe only as late as core_initcall(). I added Masami and Steve on
> > CC for their thoughts on this.
> >
> > Is there some other proper fix that I am missing?
>
> Oh, I missed that the synchronize_rcu_tasks() will be involved the kprobes
> in early stage. Does the problem only exist in the synchronize_rcu_tasks()
> instead of synchronize_rcu()? If so I can just stop optimizer in early stage
> because I just want to enable kprobes in early stage, but not optprobes.
>
> Does the following patch help?
It does look to me like it would! I clearly should have asked you about
this a couple of months ago. ;-)
The proof of the pudding would be whether the powerpc guys can apply
this to v5.10-rc7 and have their kernel come up without hanging at boot.
Thanx, Paul
> >From e5fafcda3ff918cd52619f795a3f22fb95c72b11 Mon Sep 17 00:00:00 2001
> From: Masami Hiramatsu <mhiramat@xxxxxxxxxx>
> Date: Wed, 17 Feb 2021 23:35:20 +0900
> Subject: [PATCH] kprobes: Fix to delay the kprobes jump optimization
>
> Since the kprobes jump optimization involves synchronize_rcu_tasks()
> which depends on the ksoftirqd, that can not be enabled at the
> early_initcall() boot stage. So this makes the kprobe optimization
> disabled in the early_initcall() and enables it in subsys_initcall().
>
> Note that non-optimized kprobes is still available after
> early_initcall(). Only jump optimization is delayed.
>
> Fixes: 36dadef23fcc ("kprobes: Init kprobes in early_initcall")
> Signed-off-by: Masami Hiramatsu <mhiramat@xxxxxxxxxx>
> ---
> kernel/kprobes.c | 31 +++++++++++++++++++++----------
> 1 file changed, 21 insertions(+), 10 deletions(-)
>
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index d5a3eb74a657..779d8322e307 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -861,7 +861,6 @@ static void try_to_optimize_kprobe(struct kprobe *p)
> cpus_read_unlock();
> }
>
> -#ifdef CONFIG_SYSCTL
> static void optimize_all_kprobes(void)
> {
> struct hlist_head *head;
> @@ -887,6 +886,7 @@ static void optimize_all_kprobes(void)
> mutex_unlock(&kprobe_mutex);
> }
>
> +#ifdef CONFIG_SYSCTL
> static void unoptimize_all_kprobes(void)
> {
> struct hlist_head *head;
> @@ -2497,18 +2497,14 @@ static int __init init_kprobes(void)
> }
> }
>
> -#if defined(CONFIG_OPTPROBES)
> -#if defined(__ARCH_WANT_KPROBES_INSN_SLOT)
> - /* Init kprobe_optinsn_slots */
> - kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
> -#endif
> - /* By default, kprobes can be optimized */
> - kprobes_allow_optimization = true;
> -#endif
> -
> /* By default, kprobes are armed */
> kprobes_all_disarmed = false;
>
> +#if defined(CONFIG_OPTPROBES) && defined(__ARCH_WANT_KPROBES_INSN_SLOT)
> + /* Init kprobe_optinsn_slots for allocation */
> + kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
> +#endif
> +
> err = arch_init_kprobes();
> if (!err)
> err = register_die_notifier(&kprobe_exceptions_nb);
> @@ -2523,6 +2519,21 @@ static int __init init_kprobes(void)
> }
> early_initcall(init_kprobes);
>
> +#if defined(CONFIG_OPTPROBES)
> +static int __init init_optprobes(void)
> +{
> + /*
> + * Enable kprobe optimization - this kicks the optimizer which
> + * depends on synchronize_rcu_tasks() and ksoftirqd, that is
> + * not spawned in early initcall. So delay the optimization.
> + */
> + optimize_all_kprobes();
> +
> + return 0;
> +}
> +subsys_initcall(init_optprobes);
> +#endif
> +
> #ifdef CONFIG_DEBUG_FS
> static void report_probe(struct seq_file *pi, struct kprobe *p,
> const char *sym, int offset, char *modname, struct kprobe *pp)
> --
> 2.25.1
>
>
> --
> Masami Hiramatsu <mhiramat@xxxxxxxxxx>
