On Tue, Jan 17, 2023 at 08:37:16AM +0800, Zhouyi Zhou wrote: > On Tue, Jan 17, 2023 at 8:15 AM Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote: > > > > On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote: > > > Hi Zhouyi, > > > > > > On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> wrote: > > > > > > > [..] > > > > On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote: > > > > > > > > > > Hello, > > > > > I am seeing -EBUSY returned a lot during torture_onoff() when running > > > > > rcutorture on arm64. This causes hotplug failure 30% of the time. I am > > > > > also seeing this in 6.1-rc kernels. I believe see this only for CPU0. > > > > > > > > > > This causes warnings in torture tests: > > > > > [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > > > > > [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16 > > > > > > > > > > Full kernel log here: > > > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log > > > > > > > > > > Any ideas on why this is happening and only for CPU 0 (presumably the > > > > > boot CPU)? I'd personally need these warnings to go away for my tests > > > > > as this causes rcutorture's tests to not cleanly pass for me. It > > > > > appears remove_cpu() -> device_offline() is what returns the error. > > > > > > > > > I guess this probably because CPU 0 is the tick_do_timer_cpu in > > > > nohz_full mode, which prevent that cpu from > > > > going offline [1]. We have discussed this topic, but there is no > > > > agreement on how to solve it yet. > > > > > > But I am seeing the issue in TRACE02 config which is: > > > CONFIG_NO_HZ_IDLE=y > > > # CONFIG_NO_HZ_FULL is not set > > > > > > So that is not NO_HZ_FULL: > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/ > > > However, I can't seem to find the full kernel logs for that. > > > > > > Also, other than the TRACE02 fail, I only see the issue with configs > > > with CONFIG_NO_HZ_FULL=y > > > > > > Can you try TRACE02 specifically, and see if you can reproduce the > > > same issue on your setup? Meanwhile, I'll try to trace what is > > > returning the -EBUSY. > I am trying TRACE02 on my X86_64 machine using cross compile and > qemu-system-aarch64 now, my equipment is limited, but hope I can be of > beneficial to the community ;-) Cool, I am assuming you are trying the patch you shared which you wrote in November. I bet you will still see the issue. > > > > How about something simple like the following? (untested) > > > > ---8<----------------------- > > > > diff --git a/kernel/torture.c b/kernel/torture.c > > index bc8fb361efc0..cd64110694c0 100644 > > --- a/kernel/torture.c > > +++ b/kernel/torture.c > > @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, > > // PCI probe frequently disables hotplug during boot. > > (*n_offl_attempts)--; > > s = " (-EBUSY forgiven during boot)"; > > + } else if (tick_nohz_full_running && ret == -EBUSY) { > > + (*n_offl_attempts)--; > > + s = " (-EBUSY forgiven if nohz_full is running)"; > Fantastic fix!! thus we can fix the time keeper cpu torture problem > without touch the time keeper code. Thanks. Unfortunately this does not fix the issue for TRACE02 and the patch you shared does not fix it either -- because TRACE02 is not a no-hz-full test. :-( We will need to do a bit of tracing to figure out where the -EBUSY is coming from for TRACE02. I wonder if we should ignore -EBUSY altogether, since as Thomas mentioned, hotplug failure is "normal". Thoughts? thanks, - Joel
