On Wed, Jan 25, 2023 at 8:13 AM Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> wrote: > > On Wed, Jan 25, 2023 at 6:56 AM Paul E. McKenney <paulmck@xxxxxxxxxx> wrote: > > > > On Tue, Jan 24, 2023 at 05:31:26PM +0000, Joel Fernandes (Google) wrote: > > > For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. > > > However, cpu_is_hotpluggable() still returns true for those CPUs. This causes > > > torture tests that do offlining to end up trying to offline this CPU causing > > > test failures. Such failure happens on all architectures. > > > > > > Fix it by asking the opinion of the nohz subsystem on whether the CPU can > > > be hotplugged. > > > > > > [ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ] > > > > > > For drivers/base/ portion: > > > Acked-by: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx> > > > > > > Cc: Frederic Weisbecker <frederic@xxxxxxxxxx> > > > Cc: "Paul E. McKenney" <paulmck@xxxxxxxxxx> > > > Cc: Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> > > > Cc: Will Deacon <will@xxxxxxxxxx> > > > Cc: Marc Zyngier <maz@xxxxxxxxxx> > > > Cc: rcu <rcu@xxxxxxxxxxxxxxx> > > > Cc: stable@xxxxxxxxxxxxxxx > > > Fixes: 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel") > > > Signed-off-by: Joel Fernandes (Google) <joel@xxxxxxxxxxxxxxxxx> > > > > Queued for further review and testing, thank you both! > > > > It might be a few hours until it becomes publicly visible, but it will > > get there. > A new round of rcutorture test on fixed linux-5.15.y[3] has been > performed in the PPC VM of Open Source Lab of Oregon State University > [1], which will last about 29 hours. The test result on original > linux-5.15.y is at [2]. >From the result of [1], the HOTPLUG failure reports have been eliminated, but a new kernel null point bug related to scsi module has been reported [4] ;-( [ 5.178733][ C1] BUG: Kernel NULL pointer dereference on read at 0x00000008 ... [ 5.231013][ C1] [c00000001ff9fca0] [c0000000009ffbc8] scsi_end_request+0xd8/0x1f0 (unreliable)^M [ 5.234961][ C1] [c00000001ff9fcf0] [c000000000a00e68] scsi_io_completion+0x88/0x700^M [ 5.237863][ C1] [c00000001ff9fda0] [c0000000009f5028] scsi_finish_command+0xe8/0x150^M [ 5.240089][ C1] [c00000001ff9fdf0] [c000000000a00c70] scsi_complete+0x90/0x140^M [ 5.242481][ C1] [c00000001ff9fe20] [c0000000007e5170] blk_complete_reqs+0x80/0xa0^M [ 5.245187][ C1] [c00000001ff9fe50] [c000000000f0b5d0] __do_softirq+0x1e0/0x4e0^M [ 5.248479][ C1] [c00000001ff9ff90] [c0000000000170e8] do_softirq_own_stack+0x48/0x60^M [ 5.250919][ C1] [c00000000a5e7c40] [c00000000a5e7c80] 0xc00000000a5e7c80^M [ 5.253792][ C1] [c00000000a5e7c70] [c0000000001534c0] do_softirq+0xb0/0xc0^M [ 5.256824][ C1] [c00000000a5e7ca0] [c0000000001535ac] __local_bh_enable_ip+0xdc/0x110^M [ 5.259414][ C1] [c00000000a5e7cc0] [c0000000001d75e8] irq_forced_thread_fn+0xc8/0xf0^M [ 5.261921][ C1] [c00000000a5e7d00] [c0000000001d7ae4] irq_thread+0x1b4/0x2a0^M [ 5.265298][ C1] [c00000000a5e7da0] [c00000000017d8c8] kthread+0x1a8/0x1d0^M [ 5.269184][ C1] [c00000000a5e7e10] [c00000000000cee4] ret_from_kernel_thread+0x5c/0x64^M But when I invoked [5] by hand, the bug did not show again [6]. [4] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.log [5] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/qemu-cmd [6] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/results-clocksourcewd-2/TREE03/console.zzy.log I think the bug is not caused by Joel's patch, there must be some other reason. I am starting the 29 hours' rcutorture test again. And I can give ssh access to you if you are interested in it. Sorry for the inconvenience that I bring Thanks Zhouyi > > [1] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.25-00.04.30-torture/ > [2] http://140.211.169.189/linux-stable-rc/tools/testing/selftests/rcutorture/res/2023.01.18-13.22.39-torture/ > [3] http://140.211.169.189/linux-stable-rc/ > > Happy to benefit the community and this is a fruitful learning journey ;-) > > Cheers > Zhouyi > > > > Thanx, Paul > > > > > --- > > > Sorry, resending with CC to stable. > > > > > > drivers/base/cpu.c | 3 ++- > > > include/linux/tick.h | 2 ++ > > > kernel/time/tick-sched.c | 11 ++++++++--- > > > 3 files changed, 12 insertions(+), 4 deletions(-) > > > > > > diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c > > > index 55405ebf23ab..450dca235a2f 100644 > > > --- a/drivers/base/cpu.c > > > +++ b/drivers/base/cpu.c > > > @@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = { > > > bool cpu_is_hotpluggable(unsigned int cpu) > > > { > > > struct device *dev = get_cpu_device(cpu); > > > - return dev && container_of(dev, struct cpu, dev)->hotpluggable; > > > + return dev && container_of(dev, struct cpu, dev)->hotpluggable > > > + && tick_nohz_cpu_hotpluggable(cpu); > > > } > > > EXPORT_SYMBOL_GPL(cpu_is_hotpluggable); > > > > > > diff --git a/include/linux/tick.h b/include/linux/tick.h > > > index bfd571f18cfd..9459fef5b857 100644 > > > --- a/include/linux/tick.h > > > +++ b/include/linux/tick.h > > > @@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk, > > > enum tick_dep_bits bit); > > > extern void tick_nohz_dep_clear_signal(struct signal_struct *signal, > > > enum tick_dep_bits bit); > > > +extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu); > > > > > > /* > > > * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases > > > @@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { } > > > > > > static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { } > > > static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { } > > > +static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; } > > > > > > static inline void tick_dep_set(enum tick_dep_bits bit) { } > > > static inline void tick_dep_clear(enum tick_dep_bits bit) { } > > > diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c > > > index 9c6f661fb436..63e3e8ebcd64 100644 > > > --- a/kernel/time/tick-sched.c > > > +++ b/kernel/time/tick-sched.c > > > @@ -510,7 +510,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask) > > > tick_nohz_full_running = true; > > > } > > > > > > -static int tick_nohz_cpu_down(unsigned int cpu) > > > +bool tick_nohz_cpu_hotpluggable(unsigned int cpu) > > > { > > > /* > > > * The tick_do_timer_cpu CPU handles housekeeping duty (unbound > > > @@ -518,8 +518,13 @@ static int tick_nohz_cpu_down(unsigned int cpu) > > > * CPUs. It must remain online when nohz full is enabled. > > > */ > > > if (tick_nohz_full_running && tick_do_timer_cpu == cpu) > > > - return -EBUSY; > > > - return 0; > > > + return false; > > > + return true; > > > +} > > > + > > > +static int tick_nohz_cpu_down(unsigned int cpu) > > > +{ > > > + return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY; > > > } > > > > > > void __init tick_nohz_init(void) > > > -- > > > 2.39.1.405.gd4c25cc71f-goog > > >