I looked into Linux source code and here make an attempt to answer my question:
Linux uses `root_domain` to manage bandwidth and uses `cpuset` to
manage cpu mask. Overlapping cpusets are unioned into one root
domain. So each CPU belongs to only one `root_domain` but may belong
to multiple `cpuset`. When the cpuset is a subset of the
`root_domain`, it rejects new tasks because of the
not-containing-all-cpu rule. It makes sense because the acceptance
test is performed against the bandwidth, which is defined per root
domain. This is shown by scenario 3, where the root domain contains
cpus 0,1,2 but either cpuset contains only two of the three cpus, so
adding tasks fails.
For scenario 1, when a task is added to A, it's added to the root
domain of {0,1} or {2,3}, depending on where the task executes on. So
allowed total utilization or bandwidth is two.
Please correct me if any of this is wrong.
Thanks,
Ming
On Mon, Jun 24, 2019 at 4:17 PM Ming Yang <minos.future@xxxxxxxxx> wrote:
>
> Hi,
>
> On Fri, Jun 21, 2019 at 3:34 AM Luca Abeni <lucabe72@xxxxxxxxx> wrote:
> >
> > Hi,
> >
> > On Fri, 21 Jun 2019 at 02:54, Ming Yang <minos.future@xxxxxxxxx> wrote:
> > [...]
> > > I have a question about the admission test with cpuset. Is the
> > > sched_deadline admission test examined with respect to the CPUs
> > > allowed in the cpuset?
> >
> > Yes, in theory you should be able to use up to 95% (or whatever you
> > configured in /proc/sys/kernel/sched_rt*) of the CPUs contained in the
> > cpuset.
> >
> > I am pretty sure it worked correctly when I tested this, but it was a
> > lot of time ago...
> >
>
> Thanks!
> I suppose acceptance test on cpusets that share CPUs doesn't work? For
> example, the following three scenarios:
>
> Scenario 1, sub-cpusets:
> cpuset/A/cpuset.cpu_exclusive: 1
> cpuset/A/cpuset.sched_load_balance: 0
> cpuset/A/cpuset.cpus: 0,1,2,3
> cpuset/A/B/cpuset.cpu_exclusive: 1
> cpuset/A/B/cpuset.cpus: 0,1
> cpuset/A/C/cpuset.cpu_exclusive: 1
> cpuset/A/C/cpuset.cpus: 2,3
> Is any acceptance test examined when a task is added to A or B (or C)?
> My experiment results:
> For A, yes, but it fails to utilize more than *two* cores when tasks
> are added to cpuset/A/tasks. Even if all cores are added to
> A/cpuset.cpus, it's still examined against two cores.
> For B (or C), yes, the acceptance test is examined for total
> utilization of two cores when tasks are added to cpuset/A/B(or
> C)/tasks.
>
> Scenario 2, identical cpusets:
> cpuset/A/cpuset.cpu_exclusive: 0
> cpuset/A/cpuset.cpus: 0,1
> cpuset/B/cpuset.cpu_exclusive: 0
> cpuset/B/cpuset.cpus: 0,1
> Is any acceptance test examined when a task is added to A (or B)?
> My experiment result:
> Yes, the acceptance test is examined with respect to cpuset {0,1},
> regardless of A or B.
>
> Scenario 3, overlapping cpusets:
> cpuset/A/cpuset.cpu_exclusive: 0
> cpuset/A/cpuset.cpus: 0,1
> cpuset/B/cpuset.cpu_exclusive: 0
> cpuset/B/cpuset.cpus: 1,2
> Is any acceptance test examined when a task is added to A (or B)?
> My experiment result:
> Operation not permitted for adding deadline tasks to cpusets that
> overlaps with others.
>
> > > I get deterministic admission test result without cpuset. m-1 cores
> > > can be utilized (not m?).
> >
> > The maximum SCHED_DEADLINE utilization should be 95% of m (unless you
> > changed the values in /proc/sys/kernel/sched_rt_{runtime,period}_us).
> >
> >
> > > But with cpuset, the total utilization of accepted sched_deadline
> > > tasks seems not respecting the cpuset and changes every run. For
> > > example, if cpuset contains 4 cores, I can have task sets (bound to
> > > this cpuset) utilizing 400%, 600%, or 300% before failing to add
> > > another task. It could be my program's issue. I'm still looking into
> > > it.
> >
> > If you get this result, I suspect you found a bug... Can you provide a
> > simple testcase for reproducing int?
> >
>
> I'll try to send you a test case soon.
> In the meantime, I figured out why I got the EPERM I mentioned in my
> first email. It's because I didn't disable cpuset.sched_load_balance
> at the root domain. It makes sense as allowing tasks to move in and
> out the cpuset violates the acceptance test assumption. But it might
> worth emphasizing this flag in the documentation though.
>
> Thanks,
> Ming
>
> >
> > Luca
