> On Jan 16, 2023, at 10:15 PM, Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> wrote: > > On Tue, Jan 17, 2023 at 9:45 AM Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote: >> >>> On Tue, Jan 17, 2023 at 08:37:16AM +0800, Zhouyi Zhou wrote: >>> On Tue, Jan 17, 2023 at 8:15 AM Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote: >>>> >>>> On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote: >>>>> Hi Zhouyi, >>>>> >>>>> On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi@xxxxxxxxx> wrote: >>>>>> >>>>> [..] >>>>>> On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> I am seeing -EBUSY returned a lot during torture_onoff() when running >>>>>>> rcutorture on arm64. This causes hotplug failure 30% of the time. I am >>>>>>> also seeing this in 6.1-rc kernels. I believe see this only for CPU0. >>>>>>> >>>>>>> This causes warnings in torture tests: >>>>>>> [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16 >>>>>>> [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16 >>>>>>> >>>>>>> Full kernel log here: >>>>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log >>>>>>> >>>>>>> Any ideas on why this is happening and only for CPU 0 (presumably the >>>>>>> boot CPU)? I'd personally need these warnings to go away for my tests >>>>>>> as this causes rcutorture's tests to not cleanly pass for me. It >>>>>>> appears remove_cpu() -> device_offline() is what returns the error. >>>>>>> >>>>>> I guess this probably because CPU 0 is the tick_do_timer_cpu in >>>>>> nohz_full mode, which prevent that cpu from >>>>>> going offline [1]. We have discussed this topic, but there is no >>>>>> agreement on how to solve it yet. >>>>> >>>>> But I am seeing the issue in TRACE02 config which is: >>>>> CONFIG_NO_HZ_IDLE=y >>>>> # CONFIG_NO_HZ_FULL is not set >>>>> >>>>> So that is not NO_HZ_FULL: >>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/ >>>>> However, I can't seem to find the full kernel logs for that. >>>>> >>>>> Also, other than the TRACE02 fail, I only see the issue with configs >>>>> with CONFIG_NO_HZ_FULL=y >>>>> >>>>> Can you try TRACE02 specifically, and see if you can reproduce the >>>>> same issue on your setup? Meanwhile, I'll try to trace what is >>>>> returning the -EBUSY. >>> I am trying TRACE02 on my X86_64 machine using cross compile and >>> qemu-system-aarch64 now, my equipment is limited, but hope I can be of >>> beneficial to the community ;-) >> >> Cool, I am assuming you are trying the patch you shared which you wrote in >> November. I bet you will still see the issue. > yes, I still see the issue with no hz full. >> >>>> >>>> How about something simple like the following? (untested) >>>> >>>> ---8<----------------------- >>>> >>>> diff --git a/kernel/torture.c b/kernel/torture.c >>>> index bc8fb361efc0..cd64110694c0 100644 >>>> --- a/kernel/torture.c >>>> +++ b/kernel/torture.c >>>> @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, >>>> // PCI probe frequently disables hotplug during boot. >>>> (*n_offl_attempts)--; >>>> s = " (-EBUSY forgiven during boot)"; >>>> + } else if (tick_nohz_full_running && ret == -EBUSY) { >>>> + (*n_offl_attempts)--; >>>> + s = " (-EBUSY forgiven if nohz_full is running)"; >>> Fantastic fix!! thus we can fix the time keeper cpu torture problem >>> without touch the time keeper code. >> >> Thanks. Unfortunately this does not fix the issue for TRACE02 and the patch >> you shared does not fix it either -- because TRACE02 is not a no-hz-full >> test. :-( >> >> We will need to do a bit of tracing to figure out where the -EBUSY is coming >> from for TRACE02. > agree TRACE02 is another issue, unfortunately I can't reproduce the > bug neither with your original Image [1] > nor with my cross compiled kernel using [2]. > > I guess there may be two reasons: > 1) my testbed is X86_64 based. > 2) the command that I invoke qemu is not right: > 2-1) the newly compiled linux-5.15.89-rc1 > qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp 4 Does 8 CPUs make any difference? That is my setup. Not sure what else is different. It could be a CPU model specific issue, or something. But why donot you just use the same setup you used in November and check TRACE02? That is actually what I was requesting you to rest, since you saw the same issue on that setup. Thanks, Joel > -serial file:/tmp/consoleJan1702.log -kernel arch/arm64/boot/Image > -append "console=ttyAMA0 oops=panic panic_on_warn=1 panic=-1 > ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ > rcutorture.torture_type=tasks-tracing rcutorture.onoff_interval=1000 > rcutorture.onoff_holdoff=1000 rcutorture.n_barrier_cbs=4 > rcutorture.stat_interval=15 rcutorture.shutdown_secs=1200 > test_no_idle_hz=1 verbose=1" -m 2048 -net user,hostfwd=tcp::10024-:22 > -net nic > 2-2) original Image [1] > qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp > 4 -serial file:/tmp/consoleJan1701.log -kernel /home/zzy/Image > -append "console=ttyAMA0 oops=panic panic_on_warn=1 panic=-1 > ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ > rcutorture.torture_type=tasks-tracing rcutorture.onoff_interval=1000 > rcutorture.onoff_holdoff=30 n_barrier_cbs=4 > rcutorture.stat_interval=15 rcutorture.shutdown_secs=1200 > test_no_idle_hz=1 verbose=1" -m 2048 -net > user,hostfwd=tcp::10023-:22 -net nic > > As Mark can reproduce the issue using [1], there must be something > wrong with my x86_64 based environment. > > Sorry not to be of help this time. > > I am very happy and interested to perform further tests whenever there > are further instructions ;-) > > [1] http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/Image > [2] http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/.config >> >> I wonder if we should ignore -EBUSY altogether, since as Thomas mentioned, >> hotplug failure is "normal". Thoughts? > This decision is too important for a beginner like me, however may > thanks for your trust in me ;-) What does Paul think about it ;-) > > Thanks > Zhouyi >> >> thanks, >> >> - Joel >>
