On Mon, Jan 16, 2023 at 11:36:57PM -0500, Joel Fernandes wrote: > > On Jan 16, 2023, at 11:30 PM, Paul E. McKenney <paulmck@xxxxxxxxxx> wrote: > > On Tue, Jan 17, 2023 at 12:15:07AM +0000, Joel Fernandes wrote: > >>> On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote: > >>> Hi Zhouyi, > >>> > >>> On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> wrote: > >>>> > >>> [..] > >>>> On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote: > >>>>> > >>>>> Hello, > >>>>> I am seeing -EBUSY returned a lot during torture_onoff() when running > >>>>> rcutorture on arm64. This causes hotplug failure 30% of the time. I am > >>>>> also seeing this in 6.1-rc kernels. I believe see this only for CPU0. > >>>>> > >>>>> This causes warnings in torture tests: > >>>>> [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > >>>>> [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > >>>>> > >>>>> Full kernel log here: > >>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log > >>>>> > >>>>> Any ideas on why this is happening and only for CPU 0 (presumably the > >>>>> boot CPU)? I'd personally need these warnings to go away for my tests > >>>>> as this causes rcutorture's tests to not cleanly pass for me. It > >>>>> appears remove_cpu() -> device_offline() is what returns the error. > >>>>> > >>>> I guess this probably because CPU 0 is the tick_do_timer_cpu in > >>>> nohz_full mode, which prevent that cpu from > >>>> going offline [1]. We have discussed this topic, but there is no > >>>> agreement on how to solve it yet. > >>> > >>> But I am seeing the issue in TRACE02 config which is: > >>> CONFIG_NO_HZ_IDLE=y > >>> # CONFIG_NO_HZ_FULL is not set > >>> > >>> So that is not NO_HZ_FULL: > >>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/ > >>> However, I can't seem to find the full kernel logs for that. > >>> > >>> Also, other than the TRACE02 fail, I only see the issue with configs > >>> with CONFIG_NO_HZ_FULL=y > >>> > >>> Can you try TRACE02 specifically, and see if you can reproduce the > >>> same issue on your setup? Meanwhile, I'll try to trace what is > >>> returning the -EBUSY. > >> > >> How about something simple like the following? (untested) > >> > >> ---8<----------------------- > >> > >> diff --git a/kernel/torture.c b/kernel/torture.c > >> index bc8fb361efc0..cd64110694c0 100644 > >> --- a/kernel/torture.c > >> +++ b/kernel/torture.c > >> @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, > >> // PCI probe frequently disables hotplug during boot. > >> (*n_offl_attempts)--; > >> s = " (-EBUSY forgiven during boot)"; > >> + } else if (tick_nohz_full_running && ret == -EBUSY) { > >> + (*n_offl_attempts)--; > >> + s = " (-EBUSY forgiven if nohz_full is running)"; > > > > But this should be forgiven for the timekeeping CPU, not everyone, > > correct? > > > > Yes, I know that CPU-hotplug operations can fail, but in my testing > > they almost never do. This means that a new failure might well be a > > real bug somewhere that needs attention. > > Sure. We may need to expose some API to reveal that. > > It appeared though that Thomas in the other thread related to patch > from Zhouyi, was suggesting that rcutorture tolerate hotplug failure > though, because they are not abnormal, right? Based on my rcutorture testing experience on x86, they are not at all normal. The only time I have seen rcutorture CPU-hotplug failures has been due to some bug that needed fixing. Is there a plan to make CPU hotplug failures more frequent? Thanx, Paul
