On 19/02/25 19:14, Dietmar Eggemann wrote:
> On 19/02/2025 14:09, Dietmar Eggemann wrote:
> > On 19/02/2025 11:02, Juri Lelli wrote:
> >> On 19/02/25 10:29, Dietmar Eggemann wrote:
>
> [...]
>
> >> If you already had the patch ignoring sugovs bandwidth in your set, could
> >> you please share the full dmesg?
> >
> > Will do later today ... busy with other stuff right now ;-(
>
> l B B l l l
> ^^^
> isolcpus=[3,4]
>
> w/o sugov tasks:
>
> The issue seems to be that we call partition_sched_domains() for CPU4
> during suspend. Which does not issue a:
>
> build_sched_domains() -> cpu_attach_domain() -> rq_attach_root() ->
> __dl_server_[de|at]tach_root()
And unfortunately this is the path that I am not able to reproduce at my
end. Looks like the boxes I have access to don't use this method when
suspending (no hotplug).
> [ 171.006436] dl_bw_deactivate() called cpu=4
> ...
> [ 171.006639] __dl_overflow() dl_b->bw=996147 cap=446 cap_scale(dl_b->bw, cap)=433868 dl_b->total_bw=104856 old_bw=52428 new_bw=0 type=DEF rd->span=3-5
> ^^^^^^^^^^^^^^^^^^^^^(*)
> [ 171.006652] dl_bw_manage() cpu=4 cap=446 overflow=0 req=0 return=0 type=DEF
> ...
> [ 171.007971] dl_bw_deactivate() called cpu=3
> ...
> [ 171.007999] dl_bw_manage: cpu=3 cap=0 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=1 type=DEF span=3-5
> ^^^^^^^^^^^^^^^ (*)
> [ 171.008010] dl_bw_cpus() cpu=3 rd->span=3-5 cpu_active_mask=0-3 cpus=1 type=DEF
> [ 171.008019] dl_bw_manage() cpu=3 cap=0 overflow=1 req=0 return=-16 type=DEF
> [ 171.008069] Error taking CPU3 down: -16
>
> You can see how 'dl_b->total_bw' stays 104856 (2 x util = 51) even
> though CPU4 is off (*).
Right (well, not right :).
> If total_bw would be 52428 for CPU3 going down we would still fail with
> the current code (taking the else path):
>
> 3604 if (dl_bw_cpus(cpu) - 1)
> 3605 overflow = __dl_overflow(dl_b, cap, fair_server_bw, 0);
> 3606 else
> 3607 overflow = 1;
>
> but if we would take the if path even when 'dl_bw_cpus(cpu) = 1'
> _dl_overflow() would return false:
>
> 280 return dl_b->bw != -1 &&
> 281 cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
>
> '0 < 52428 - 52428 + 0' is false
OK. The idea of the current logic is that we shouldn't even enter that
branch, as, if total_bw accounting was correct, total_bw for DEF would
be equal to fair_server_bw. So, no additional DEADLINE bandwidth present
and we proceed with offlining.
Let's have a look below.
> ---
>
> [ 170.847396] PM: suspend entry (deep)
> [ 170.852093] Filesystems sync: 0.000 seconds
> [ 170.859274] Freezing user space processes
> [ 170.864616] Freezing user space processes completed (elapsed 0.001 seconds)
> [ 170.871614] OOM killer disabled.
> [ 170.874861] Freezing remaining freezable tasks
> [ 170.880499] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
> [ 170.887936] printk: Suspending console(s) (use no_console_suspend to debug)
> [ 171.000031] arm-scmi arm-scmi.1.auto: timed out in resp(caller: do_xfer+0x114/0x494)
> [ 171.001421] Disabling non-boot CPUs ...
CPU5 going offline.
> [ 171.001501] dl_bw_deactivate() called cpu=5
> [ 171.001518] __dl_bw_capacity() mask=0-2,5 cap=2940
> [ 171.001530] dl_bw_cpus() cpu=5 rd->span=0-2,5 cpu_active_mask=0-5 cpumask_weight(rd->span)=4 type=DYN
> [ 171.001541] dl_bw_manage: cpu=5 cap=2494 fair_server_bw=52428 total_bw=209712 dl_bw_cpus=4 type=DYN span=0-2,5
> [ 171.001553] dl_bw_cpus() cpu=5 rd->span=0-2,5 cpu_active_mask=0-5 cpumask_weight(rd->span)=4 type=DYN
> [ 171.001567] CPU: 5 UID: 0 PID: 41 Comm: cpuhp/5 Not tainted 6.13.0-09343-g9ce523149e08-dirty #172
> [ 171.001578] Hardware name: ARM Juno development board (r0) (DT)
> [ 171.001583] Call trace:
> [ 171.001587] show_stack+0x18/0x24 (C)
> [ 171.001605] dump_stack_lvl+0x74/0x8c
> [ 171.001621] dump_stack+0x18/0x24
> [ 171.001634] dl_bw_manage+0x3a0/0x500
> [ 171.001650] dl_bw_deactivate+0x40/0x50
> [ 171.001661] sched_cpu_deactivate+0x34/0x24c
> [ 171.001676] cpuhp_invoke_callback+0x138/0x694
> [ 171.001689] cpuhp_thread_fun+0xb0/0x198
> [ 171.001702] smpboot_thread_fn+0x200/0x224
> [ 171.001715] kthread+0x12c/0x204
> [ 171.001727] ret_from_fork+0x10/0x20
> [ 171.001741] __dl_overflow() dl_b->bw=996147 cap=2494 cap_scale(dl_b->bw, cap)=2426162 dl_b->total_bw=209712 old_bw=52428 new_bw=0 type=DYN rd->span=0-2,5
> [ 171.001754] dl_bw_manage() cpu=5 cap=2494 overflow=0 req=0 return=0 type=DYN
> [ 171.001814] partition_sched_domains() called
> [ 171.001821] CPU: 5 UID: 0 PID: 41 Comm: cpuhp/5 Not tainted 6.13.0-09343-g9ce523149e08-dirty #172
> [ 171.001831] Hardware name: ARM Juno development board (r0) (DT)
> [ 171.001835] Call trace:
> [ 171.001838] show_stack+0x18/0x24 (C)
> [ 171.001849] dump_stack_lvl+0x74/0x8c
> [ 171.001862] dump_stack+0x18/0x24
> [ 171.001875] partition_sched_domains+0x48/0x7c
> [ 171.001886] sched_cpu_deactivate+0x1a8/0x24c
> [ 171.001900] cpuhp_invoke_callback+0x138/0x694
> [ 171.001913] cpuhp_thread_fun+0xb0/0x198
> [ 171.001925] smpboot_thread_fn+0x200/0x224
> [ 171.001937] kthread+0x12c/0x204
> [ 171.001948] ret_from_fork+0x10/0x20
> [ 171.001961] partition_sched_domains_locked() ndoms_new=1
> [ 171.002012] cpu_attach_domain() called cpu=0 type=DEF
> [ 171.002018] CPU0 attaching NULL sched-domain.
> [ 171.002022] span=3-4
> [ 171.002029] rq_attach_root() called cpu=0 type=DEF
> [ 171.002043] dl_bw_cpus() cpu=0 rd->span=0-2,5 cpu_active_mask=0-4 cpus=3 type=DYN
> [ 171.002053] __dl_server_detach_root() called cpu=0
> [ 171.002059] dl_bw_cpus() cpu=0 rd->span=0-2,5 cpu_active_mask=0-4 cpus=3 type=DYN
> [ 171.002068] __dl_sub() tsk_bw=52428 dl_b->total_bw=157284 type=DYN rd->span=0-2,5
> [ 171.002077] __dl_update() (3) cpu=0 rq->dl.extra_bw=603812
> [ 171.002083] __dl_update() (3) cpu=1 rq->dl.extra_bw=869446
> [ 171.002089] __dl_update() (3) cpu=2 rq->dl.extra_bw=1013623
> [ 171.002098] dl_bw_cpus() cpu=0 rd->span=0,3-4 cpu_active_mask=0-4 cpumask_weight(rd->span)=3 type=DEF
> [ 171.002109] __dl_server_attach_root() called cpu=0
> [ 171.002114] dl_bw_cpus() cpu=0 rd->span=0,3-4 cpu_active_mask=0-4 cpumask_weight(rd->span)=3 type=DEF
> [ 171.002124] __dl_add() tsk_bw=52428 dl_b->total_bw=157284 type=DEF rd->span=0,3-4
> [ 171.002133] __dl_update() (3) cpu=0 rq->dl.extra_bw=586336
> [ 171.002139] __dl_update() (3) cpu=3 rq->dl.extra_bw=1004885
> [ 171.002145] __dl_update() (3) cpu=4 rq->dl.extra_bw=1017992
> [ 171.002153] cpu_attach_domain() called cpu=1 type=DEF
> [ 171.002159] CPU1 attaching NULL sched-domain.
> [ 171.002163] span=0,3-4
> [ 171.002169] rq_attach_root() called cpu=1 type=DEF
> [ 171.002181] dl_bw_cpus() cpu=1 rd->span=1-2,5 cpu_active_mask=0-4 cpus=2 type=DYN
> [ 171.002191] __dl_server_detach_root() called cpu=1
> [ 171.002196] dl_bw_cpus() cpu=1 rd->span=1-2,5 cpu_active_mask=0-4 cpus=2 type=DYN
> [ 171.002206] __dl_sub() tsk_bw=52428 dl_b->total_bw=104856 type=DYN rd->span=1-2,5
> [ 171.002215] __dl_update() (3) cpu=1 rq->dl.extra_bw=895660
> [ 171.002221] __dl_update() (3) cpu=2 rq->dl.extra_bw=1039837
> [ 171.002228] dl_bw_cpus() cpu=1 rd->span=0-1,3-4 cpu_active_mask=0-4 cpumask_weight(rd->span)=4 type=DEF
> [ 171.002238] __dl_server_attach_root() called cpu=1
> [ 171.002243] dl_bw_cpus() cpu=1 rd->span=0-1,3-4 cpu_active_mask=0-4 cpumask_weight(rd->span)=4 type=DEF
> [ 171.002253] __dl_add() tsk_bw=52428 dl_b->total_bw=209712 type=DEF rd->span=0-1,3-4
> [ 171.002262] __dl_update() (3) cpu=0 rq->dl.extra_bw=573229
> [ 171.002267] __dl_update() (3) cpu=1 rq->dl.extra_bw=882553
> [ 171.002273] __dl_update() (3) cpu=3 rq->dl.extra_bw=991778
> [ 171.002279] __dl_update() (3) cpu=4 rq->dl.extra_bw=1004885
> [ 171.002286] cpu_attach_domain() called cpu=2 type=DEF
> [ 171.002291] CPU2 attaching NULL sched-domain.
> [ 171.002296] span=0-1,3-4
> [ 171.002301] rq_attach_root() called cpu=2 type=DEF
> [ 171.002314] dl_bw_cpus() cpu=2 rd->span=2,5 cpu_active_mask=0-4 cpus=1 type=DYN
> [ 171.002323] __dl_server_detach_root() called cpu=2
> [ 171.002329] dl_bw_cpus() cpu=2 rd->span=2,5 cpu_active_mask=0-4 cpus=1 type=DYN
> [ 171.002338] __dl_sub() tsk_bw=52428 dl_b->total_bw=52428 type=DYN rd->span=2,5
> [ 171.002346] __dl_update() (3) cpu=2 rq->dl.extra_bw=1092265
> [ 171.002353] dl_bw_cpus() cpu=2 rd->span=0-4 cpu_active_mask=0-4 cpumask_weight(rd->span)=5 type=DEF
> [ 171.002363] __dl_server_attach_root() called cpu=2
> [ 171.002368] dl_bw_cpus() cpu=2 rd->span=0-4 cpu_active_mask=0-4 cpumask_weight(rd->span)=5 type=DEF
> [ 171.002377] __dl_add() tsk_bw=52428 dl_b->total_bw=262140 type=DEF rd->span=0-4
^^^^
OK. 5 still online CPUs dl servers temporarily added to DEF root domain.
> [ 171.002385] __dl_update() (3) cpu=0 rq->dl.extra_bw=562744
> [ 171.002391] __dl_update() (3) cpu=1 rq->dl.extra_bw=872068
> [ 171.002397] __dl_update() (3) cpu=2 rq->dl.extra_bw=1081780
> [ 171.002403] __dl_update() (3) cpu=3 rq->dl.extra_bw=981293
> [ 171.002409] __dl_update() (3) cpu=4 rq->dl.extra_bw=994400
> [ 171.002416] cpu_attach_domain() called cpu=5 type=DEF
> [ 171.002421] CPU5 attaching NULL sched-domain.
> [ 171.002425] span=0-4
> [ 171.002431] rq_attach_root() called cpu=5 type=DEF
> [ 171.002438] build_sched_domains() called cpu_map=0-2
> [ 171.002556] cpu_attach_domain() called cpu=0 type=DYN
Adding CPU5 (going offline) to DEF doesn't add its dl server
contribution (OK).
> [ 171.002565] CPU0 attaching sched-domain(s):
> [ 171.002571] domain-0: span=0-2 level=PKG
> [ 171.002583] groups: 0:{ span=0 cap=445 }, 1:{ span=1-2 cap=2045 }
> [ 171.002619] rq_attach_root() called cpu=0 type=DYN
> [ 171.002630] dl_bw_cpus() cpu=0 rd->span=0-5 cpu_active_mask=0-4 cpus=5 type=DEF
> [ 171.002639] __dl_server_detach_root() called cpu=0
> [ 171.002644] dl_bw_cpus() cpu=0 rd->span=0-5 cpu_active_mask=0-4 cpus=5 type=DEF
> [ 171.002653] __dl_sub() tsk_bw=52428 dl_b->total_bw=209712 type=DEF rd->span=0-5
> [ 171.002662] __dl_update() (3) cpu=0 rq->dl.extra_bw=573229
> [ 171.002668] __dl_update() (3) cpu=1 rq->dl.extra_bw=882553
> [ 171.002674] __dl_update() (3) cpu=2 rq->dl.extra_bw=1092265
> [ 171.002680] __dl_update() (3) cpu=3 rq->dl.extra_bw=991778
> [ 171.002686] __dl_update() (3) cpu=4 rq->dl.extra_bw=1004885
> [ 171.002693] dl_bw_cpus() cpu=0 rd->span=0 cpu_active_mask=0-4 cpumask_weight(rd->span)=1 type=DYN
> [ 171.002702] __dl_server_attach_root() called cpu=0
> [ 171.002707] dl_bw_cpus() cpu=0 rd->span=0 cpu_active_mask=0-4 cpumask_weight(rd->span)=1 type=DYN
> [ 171.002716] __dl_add() tsk_bw=52428 dl_b->total_bw=52428 type=DYN rd->span=0
> [ 171.002724] __dl_update() (3) cpu=0 rq->dl.extra_bw=520801
> [ 171.002731] cpu_attach_domain() called cpu=1 type=DYN
> [ 171.002738] CPU1 attaching sched-domain(s):
> [ 171.002743] domain-0: span=1-2 level=MC
> [ 171.002753] groups: 1:{ span=1 cap=1022 }, 2:{ span=2 cap=1023 }
> [ 171.002787] domain-1: span=0-2 level=PKG
> [ 171.002798] groups: 1:{ span=1-2 cap=2045 }, 0:{ span=0 cap=445 }
> [ 171.002831] rq_attach_root() called cpu=1 type=DYN
> [ 171.002841] dl_bw_cpus() cpu=1 rd->span=1-5 cpu_active_mask=0-4 cpus=4 type=DEF
> [ 171.002851] __dl_server_detach_root() called cpu=1
> [ 171.002856] dl_bw_cpus() cpu=1 rd->span=1-5 cpu_active_mask=0-4 cpus=4 type=DEF
> [ 171.002865] __dl_sub() tsk_bw=52428 dl_b->total_bw=157284 type=DEF rd->span=1-5
> [ 171.002873] __dl_update() (3) cpu=1 rq->dl.extra_bw=895660
> [ 171.002879] __dl_update() (3) cpu=2 rq->dl.extra_bw=1105372
> [ 171.002885] __dl_update() (3) cpu=3 rq->dl.extra_bw=1004885
> [ 171.002891] __dl_update() (3) cpu=4 rq->dl.extra_bw=1017992
> [ 171.002898] dl_bw_cpus() cpu=1 rd->span=0-1 cpu_active_mask=0-4 cpumask_weight(rd->span)=2 type=DYN
> [ 171.002907] __dl_server_attach_root() called cpu=1
> [ 171.002912] dl_bw_cpus() cpu=1 rd->span=0-1 cpu_active_mask=0-4 cpumask_weight(rd->span)=2 type=DYN
> [ 171.002922] __dl_add() tsk_bw=52428 dl_b->total_bw=104856 type=DYN rd->span=0-1
> [ 171.002930] __dl_update() (3) cpu=0 rq->dl.extra_bw=494587
> [ 171.002936] __dl_update() (3) cpu=1 rq->dl.extra_bw=869446
> [ 171.002943] cpu_attach_domain() called cpu=2 type=DYN
> [ 171.002950] CPU2 attaching sched-domain(s):
> [ 171.002954] domain-0: span=1-2 level=MC
> [ 171.002965] groups: 2:{ span=2 cap=1023 }, 1:{ span=1 cap=1022 }
> [ 171.002998] domain-1: span=0-2 level=PKG
> [ 171.003009] groups: 1:{ span=1-2 cap=2045 }, 0:{ span=0 cap=445 }
> [ 171.003043] rq_attach_root() called cpu=2 type=DYN
> [ 171.003053] dl_bw_cpus() cpu=2 rd->span=2-5 cpu_active_mask=0-4 cpus=3 type=DEF
> [ 171.003062] __dl_server_detach_root() called cpu=2
> [ 171.003067] dl_bw_cpus() cpu=2 rd->span=2-5 cpu_active_mask=0-4 cpus=3 type=DEF
> [ 171.003076] __dl_sub() tsk_bw=52428 dl_b->total_bw=104856 type=DEF rd->span=2-5
^^^^
OK. CPU3 + CPU4 (CPU5 offline).
> [ 171.003085] __dl_update() (3) cpu=2 rq->dl.extra_bw=1122848
> [ 171.003091] __dl_update() (3) cpu=3 rq->dl.extra_bw=1022361
> [ 171.003096] __dl_update() (3) cpu=4 rq->dl.extra_bw=1035468
> [ 171.003103] dl_bw_cpus() cpu=2 rd->span=0-2 cpu_active_mask=0-4 cpumask_weight(rd->span)=3 type=DYN
> [ 171.003113] __dl_server_attach_root() called cpu=2
> [ 171.003118] dl_bw_cpus() cpu=2 rd->span=0-2 cpu_active_mask=0-4 cpumask_weight(rd->span)=3 type=DYN
> [ 171.003127] __dl_add() tsk_bw=52428 dl_b->total_bw=157284 type=DYN rd->span=0-2
> [ 171.003136] __dl_update() (3) cpu=0 rq->dl.extra_bw=477111
> [ 171.003141] __dl_update() (3) cpu=1 rq->dl.extra_bw=851970
> [ 171.003147] __dl_update() (3) cpu=2 rq->dl.extra_bw=1105372
> [ 171.003188] root domain span: 0-2
> [ 171.003194] default domain span: 3-5
> [ 171.003220] rd 0-2: Checking EAS, schedutil is mandatory
> [ 171.005840] psci: CPU5 killed (polled 0 ms)
OK. DYN has (CPU0,1,2) 157284 and DEF (CPU3,4) 104856.
CPU4 going offline (it's isolated on DEF).
> [ 171.006436] dl_bw_deactivate() called cpu=4
> [ 171.006446] __dl_bw_capacity() mask=3-5 cap=892
> [ 171.006454] dl_bw_cpus() cpu=4 rd->span=3-5 cpu_active_mask=0-4 cpus=2 type=DEF
> [ 171.006464] dl_bw_manage: cpu=4 cap=446 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2 type=DEF span=3-5
> [ 171.006475] dl_bw_cpus() cpu=4 rd->span=3-5 cpu_active_mask=0-4 cpus=2 type=DEF
> [ 171.006485] CPU: 4 UID: 0 PID: 36 Comm: cpuhp/4 Not tainted 6.13.0-09343-g9ce523149e08-dirty #172
> [ 171.006495] Hardware name: ARM Juno development board (r0) (DT)
> [ 171.006499] Call trace:
> [ 171.006502] show_stack+0x18/0x24 (C)
> [ 171.006514] dump_stack_lvl+0x74/0x8c
> [ 171.006528] dump_stack+0x18/0x24
> [ 171.006541] dl_bw_manage+0x3a0/0x500
> [ 171.006554] dl_bw_deactivate+0x40/0x50
> [ 171.006564] sched_cpu_deactivate+0x34/0x24c
> [ 171.006579] cpuhp_invoke_callback+0x138/0x694
> [ 171.006591] cpuhp_thread_fun+0xb0/0x198
> [ 171.006604] smpboot_thread_fn+0x200/0x224
> [ 171.006616] kthread+0x12c/0x204
> [ 171.006627] ret_from_fork+0x10/0x20
> [ 171.006639] __dl_overflow() dl_b->bw=996147 cap=446 cap_scale(dl_b->bw, cap)=433868 dl_b->total_bw=104856 old_bw=52428 new_bw=0 type=DEF rd->span=3-5
> [ 171.006652] dl_bw_manage() cpu=4 cap=446 overflow=0 req=0 return=0 type=DEF
> [ 171.006706] partition_sched_domains() called
> [ 171.006713] CPU: 4 UID: 0 PID: 36 Comm: cpuhp/4 Not tainted 6.13.0-09343-g9ce523149e08-dirty #172
> [ 171.006722] Hardware name: ARM Juno development board (r0) (DT)
> [ 171.006727] Call trace:
> [ 171.006730] show_stack+0x18/0x24 (C)
> [ 171.006740] dump_stack_lvl+0x74/0x8c
> [ 171.006754] dump_stack+0x18/0x24
> [ 171.006767] partition_sched_domains+0x48/0x7c
> [ 171.006778] sched_cpu_deactivate+0x1a8/0x24c
> [ 171.006792] cpuhp_invoke_callback+0x138/0x694
> [ 171.006805] cpuhp_thread_fun+0xb0/0x198
> [ 171.006817] smpboot_thread_fn+0x200/0x224
> [ 171.006829] kthread+0x12c/0x204
> [ 171.006840] ret_from_fork+0x10/0x20
> [ 171.006852] partition_sched_domains_locked() ndoms_new=1
> [ 171.006861] partition_sched_domains_locked() goto match2
> [ 171.006867] rd 0-2: Checking EAS, schedutil is mandatory
> [ 171.007774] psci: CPU4 killed (polled 4 ms)
As I guess you were saying above, CPU4 contribution is not removed from
DEF.
> [ 171.007971] dl_bw_deactivate() called cpu=3
> [ 171.007981] __dl_bw_capacity() mask=3-5 cap=446
> [ 171.007989] dl_bw_cpus() cpu=3 rd->span=3-5 cpu_active_mask=0-3 cpus=1 type=DEF
> [ 171.007999] dl_bw_manage: cpu=3 cap=0 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=1 type=DEF span=3-5
^^^^
And this is now wrong. :/
> [ 171.008010] dl_bw_cpus() cpu=3 rd->span=3-5 cpu_active_mask=0-3 cpus=1 type=DEF
> [ 171.008019] dl_bw_manage() cpu=3 cap=0 overflow=1 req=0 return=-16 type=DEF
> [ 171.008069] Error taking CPU3 down: -16
> [ 171.008076] Non-boot CPUs are not disabled
> [ 171.008080] Enabling non-boot CPUs ...
Hummm.
