On Thu, Jan 14, 2021 at 09:22:54AM +0000, Christoph Lameter wrote:
> On Wed, 13 Jan 2021, Marcelo Tosatti wrote:
>
> > So as discussed, this is one possible prctl interface for
> > task isolation.
> >
> > Is this something that is desired? If not, what is the
> > proper way for the interface to be?
>
> Sure that sounds liek a good beginning but I guess we need some
> specificity on the features
>
> > +Task isolation CPU interface
> > +============================
>
> How does one do a oneshot flush of OS activities?
ret = prctl(PR_TASK_ISOLATION_REQUEST, ISOL_F_QUIESCE, 0, 0, 0);
if (ret == -1) {
perror("prctl PR_TASK_ISOLATION_REQUEST");
exit(0);
}
>
> I.e. I have a polling loop over numerous shared and I/o devices in user
> space and I want to make sure that the system is quite before I enter the
> loop.
You could configure things in two ways: with syscalls allowed or not.
Syscalls disallowed:
===================
1) Add a new isolation feature ISOL_F_BLOCK_SYSCALLS (to block certain
syscalls) along with ISOL_F_SETUP_NOTIF (to notify upon isolation
breaking):
if ((ifeat & ISOL_F_BLOCK_SYSCALLS) == ISOL_F_BLOCK_SYSCALLS) {
struct task_isolation_block_syscalls tibs = { list of
syscalls to block,
additional
parameters }
struct task_isolation_notif tis = { parameters to control
signal handling upon
isolation breaking event }
ret = prctl(PR_TASK_ISOLATION_SET, ISOL_F_SETUP_NOTIF, &tis);
if (ret != 0) { ... }
featuremask |= ISOL_F_SETUP_NOTIF;
ret = prctl(PR_TASK_ISOLATION_SET, ISOL_F_BLOCK_SYSCALLS, &tibs);
if (ret != 0) { ... }
featuremask |= ISOL_F_BLOCK_SIGNALS;
featuremask |= ISOL_F_QUIESCE;
}
This would require knowledge of the behaviour of individual system
calls, that is whether or not these syscalls cause the CPU to be a target
of interruptions (1) (while the QUIESCE / HARD / WARN division you propose
allows for coarse-grained control).
Perhaps coarse control while also allowing finer grained control
(if desired) is a useful choice?
1: for example adding free pages to per-cpu free lists.
Syscalls allowed:
=================
> In the loop itself some activities may require syscalls so they will
> potentialy cause the OS services such as timers to start again.
Or a different mode where the syscall return itself can finish
any pending activities.
> When such
> an activities is complete another quiet down call can be issued.
Although this seems more efficient (if multiple syscalls are to be
used).
> Could be implemented by setting a flag that does an action and then resets
> itself? Or the flag could be reset if a syscall that requires timers etc
> is used?
You mean to let userspace know if a certain syscall triggered a pending
action which must be finished (before "quiet mode" is entered again) ?
Sounds like a good idea.
> Features that I think may be needed:
>
> F_ISOL_QUIESCE -> quiet down now but allow all OS activities. OS
> activites reset flag
>
> F_ISOL_BAREMETAL_HARD -> No OS interruptions. Fault on syscalls that
> require such actions in the future.
Question: why BAREMETAL ?
Two comments:
1) HARD mode could also block activities from different CPUs that can
interrupt this isolated CPU (for example CPU hotplug, or increasing
per-CPU trace buffer size).
Unclear whether such blockage should be performed on:
-> Individual action basis (eg: BLOCK_CPU_HOTPLUG,
BLOCK_PERCPU_TRACEBUFFER_SIZE, ...) (which could allow
individual unblocking through a sysfs interface, for example).
Or
-> Be tied to a flag with a less implementation specific meaning such as
F_ISOL_BAREMETAL_HARD.
2) For a type of application it is the case that certain interruptions
can be tolerated, as long as they do not cross certain thresholds.
For example, one loses the flexibility to read/write MSRs
on the isolated CPUs (including performance counters,
RDT/MBM type MSRs, frequency/power statistics) by
forcing a "no interruptions" mode.
That flexibility seems to be useful (so perhaps
F_ISOL_BAREMETAL_HARD but optionally permitting
certain interruptions).
> F_ISOL_BAREMETAL_WARN -> Similar. Create a warning in the syslog when OS
> services require delayed processing etc
> but continue while resetting the flag.
Alex seems to be interested in different notification methods as well.
Thanks for the input.
