On 11/02/25 09:36, Dietmar Eggemann wrote: > On 10/02/2025 18:09, Juri Lelli wrote: > > Hi Christian, > > > > Thanks for taking a look as well. > > > > On 07/02/25 15:55, Christian Loehle wrote: > >> On 2/7/25 14:04, Jon Hunter wrote: > >>> > >>> > >>> On 07/02/2025 13:38, Dietmar Eggemann wrote: > >>>> On 07/02/2025 11:38, Jon Hunter wrote: > >>>>> > >>>>> On 06/02/2025 09:29, Juri Lelli wrote: > >>>>>> On 05/02/25 16:56, Jon Hunter wrote: > >>>>>> > >>>>>> ... > >>>>>> > >>>>>>> Thanks! That did make it easier :-) > >>>>>>> > >>>>>>> Here is what I see ... > >>>>>> > >>>>>> Thanks! > >>>>>> > >>>>>> Still different from what I can repro over here, so, unfortunately, I > >>>>>> had to add additional debug printks. Pushed to the same branch/repo. > >>>>>> > >>>>>> Could I ask for another run with it? Please also share the complete > >>>>>> dmesg from boot, as I would need to check debug output when CPUs are > >>>>>> first onlined. > >>>> > >>>> So you have a system with 2 big and 4 LITTLE CPUs (Denver0 Denver1 A57_0 > >>>> A57_1 A57_2 A57_3) in one MC sched domain and (Denver1 and A57_0) are > >>>> isol CPUs? > >>> > >>> I believe that 1-2 are the denvers (even thought they are listed as 0-1 in device-tree). > >> > >> Interesting, I have yet to reproduce this with equal capacities in isolcpus. > >> Maybe I didn't try hard enough yet. > >> > >>> > >>>> This should be easy to set up for me on my Juno-r0 [A53 A57 A57 A53 A53 A53] > >>> > >>> Yes I think it is similar to this. > >>> > >>> Thanks! > >>> Jon > >>> > >> > >> I could reproduce that on a different LLLLbb with isolcpus=3,4 (Lb) and > >> the offlining order: > >> echo 0 > /sys/devices/system/cpu/cpu5/online > >> echo 0 > /sys/devices/system/cpu/cpu1/online > >> echo 0 > /sys/devices/system/cpu/cpu3/online > >> echo 0 > /sys/devices/system/cpu/cpu2/online > >> echo 0 > /sys/devices/system/cpu/cpu4/online > >> > >> while the following offlining order succeeds: > >> echo 0 > /sys/devices/system/cpu/cpu5/online > >> echo 0 > /sys/devices/system/cpu/cpu4/online > >> echo 0 > /sys/devices/system/cpu/cpu1/online > >> echo 0 > /sys/devices/system/cpu/cpu2/online > >> echo 0 > /sys/devices/system/cpu/cpu3/online > >> (Both offline an isolcpus last, both have CPU0 online) > >> > > Could reproduce on Juno-r0: > > 0 1 2 3 4 5 > > L b b L L L > > ^^^ > isol = [3-4] so both L > > echo 0 > /sys/devices/system/cpu/cpu1/online > echo 0 > /sys/devices/system/cpu/cpu4/online > echo 0 > /sys/devices/system/cpu/cpu5/online > echo 0 > /sys/devices/system/cpu/cpu2/online - isol > echo 0 > /sys/devices/system/cpu/cpu3/online - isol > > >> The issue only triggers with sugov DL threads (I guess that's obvious, but > >> just to mention it). > > IMHO, it doesn't have to be a sugov DL task. Any DL task will do. OK, but in this case we actually want to fail. If we have allocated bandwidth for an actual DL task (not a dl server or a 'fake' sugov), we don't want to inadvertently leave it w/o bandwidth by turning CPUs off. > // on a 2. shell: > # chrt -d -T 5000000 -D 10000000 -P 16666666 -p 0 $$ > > # ps -eTo comm,pid,class | grep DLN > bash 1243 DLN > > 5000000/16666666 = 0.3, 0.3 << 10 = 307 (task util, bandwidth requirement) > > > It wasn't obvious to me at first :). So thanks for confirming. > > > >> I'll investigate some more later but wanted to share for now. > > > > So, problem actually is that I am not yet sure what we should do with > > sugovs' bandwidth wrt root domain accounting. W/o isolation it's all > > good, as it gets accounted for correctly on the dynamic domains sugov > > tasks can run on. But with isolation and sugov affected_cpus that cross > > isolation domains (e.g., one BIG one little), we can get into troubles > > not knowing if sugov contribution should fall on the DEF or DYN domain. > > # echo 0 > /sys/devices/system/cpu/cpu1/online > [ 87.402722] __dl_bw_capacity() mask=0-2,5 cap=2940 > [ 87.407551] dl_bw_cpus() cpu=1 rd->span=0-2,5 cpu_active_mask=0-5 cpumask_weight(rd->span)=4 > [ 87.416019] dl_bw_manage: cpu=1 cap=1916 fair_server_bw=52428 total_bw=524284 dl_bw_cpus=4 type=DYN span=0-2,5 > > # echo 0 > /sys/devices/system/cpu/cpu2/online > [ 95.562270] __dl_bw_capacity() mask=0,2,5 cap=1916 > [ 95.567091] dl_bw_cpus() cpu=2 rd->span=0,2,5 cpu_active_mask=0,2-5 cpumask_weight(rd->span)=3 > [ 95.575735] dl_bw_manage: cpu=2 cap=892 fair_server_bw=52428 total_bw=157284 dl_bw_cpus=3 type=DYN span=0,2,5 > > # echo 0 > /sys/devices/system/cpu/cpu5/online > [ 100.573131] __dl_bw_capacity() mask=0,5 cap=892 > [ 100.577713] dl_bw_cpus() cpu=5 rd->span=0,5 cpu_active_mask=0,3-5 cpumask_weight(rd->span)=2 > [ 100.586186] dl_bw_manage: cpu=5 cap=446 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2 type=DYN span=0,5 > > # echo 0 > /sys/devices/system/cpu/cpu3/online > [ 110.232755] __dl_bw_capacity() mask=1-5 cap=892 > [ 110.237333] dl_bw_cpus() cpu=6 rd->span=1-5 cpu_active_mask=0,3-4 cpus=2 > [ 110.244064] dl_bw_manage: cpu=3 cap=446 fair_server_bw=52428 total_bw=419428 dl_bw_cpus=2 type=DEF span=1-5 > > > # echo 0 > /sys/devices/system/cpu/cpu4/online > [ 175.870273] __dl_bw_capacity() mask=1-5 cap=446 > [ 175.874850] dl_bw_cpus() cpu=6 rd->span=1-5 cpu_active_mask=0,4 cpus=1 > [ 175.881407] dl_bw_manage: cpu=4 cap=0 fair_server_bw=52428 total_bw=367000 dl_bw_cpus=1 type=DEF span=1-5 > ^^^^^ ^^^^^^^^ > w/o/ cpu4 cap is 0! cpu0 is not part of it > ... > [ 175.897600] dl_bw_manage() cpu=4 cap=0 overflow=1 return=-16 > ^^^^^^^^^^ -EBUSY > > -bash: echo: write error: Device or resource busy > > sched_cpu_deactivate() > > dl_bw_deactivate(cpu) > > dl_bw_manage(dl_bw_req_deactivate, cpu, 0); > > return overflow ? -EBUSY : 0; > > Looks like in DEF there is no CPU capacity left but we still have 1 DLN > task with a bandwidth requirement of 307. >
