Hi Sebastian, > -----Original Message----- > From: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx> > Sent: Friday, November 12, 2021 4:37 PM > To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@xxxxxxxxxxx>; > cgroups@xxxxxxxxxxxxxxx > Cc: linux-rt-users@xxxxxxxxxxxxxxx; Schild, Henning (T RDA IOT SES-DE) > <henning.schild@xxxxxxxxxxx>; Kiszka, Jan (T RDA IOT) > <jan.kiszka@xxxxxxxxxxx>; Schmidt, Adriaan (T RDA IOT SES-DE) > <adriaan.schmidt@xxxxxxxxxxx>; Frederic Weisbecker <frederic@xxxxxxxxxx> > Subject: Re: Questions about replacing isolcpus by cgroup-v2 > > On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote: > > Dear subscribers, > Hi, > > I Cced cgroups@vger since thus question fits there better. > I Cced Frederic in case he has come clues regarding isolcpus and cgroups. Indeed. Thanks! > > > we are currently evaluating how to rework realtime tuning to use cgroup-v2 > cpusets instead of the isolcpus kernel parameter. > > Our use-case are realtime applications with rt and non-rt threads. Hereby, the > non-rt thread might create additional non-rt threads: > > > > Example (RT CPU=1, 4 CPUs): > > - Non-RT Thread (A) with default affinity 0xD (1101b) > > - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity) > > > > When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly > works: > > Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a > default affinity of 0xD. > > By that, independent of the threads' priorities, we can ensure that nothing is > scheduled on our RT cpu (except from kernel threads, etc...). > > > > During this journey, we discovered the following: > > > > Using cgroup-v2 cpusets and isolcpus together seems to be incompatible: > > When activating the cpuset controller on a cgroup (for the first time), all > default CPU affinities are reset. > > By that, also the default affinity is set to 0xFFFF..., while with isolcpus we > expect it to be (0xFFFF - isolcpus). > > This breaks the example from above, as now the non-RT thread can also be > scheduled on the RT CPU. > > > > When only using cgroup-v2, we can isolate our RT process by placing it in a > cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups. > > However, we do not know of a strategy to set a default affinity: > > Given the example above, we have no way to ensure that newly created > threads are born with an affinity of just 0x2 (without changing the application). > > > > Finally, isolcpus itself is deprecated since kernel 5.4. > > Where is this the deprecation of isolcpus announced/ written? https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html isolcpus= [KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance. [Deprecated - use cpusets instead] Format: [flag-list,]<cpu-list> > > > Questions: > > > > 1. What is the best strategy to "isolcpus" similar semantics with cgroups-v2? > > 2. Is there a way to specify the default affinity (within a cgroup) > > > > We are currently at a point where we would write patches to add a default > affinity feature to cpusets of cgroupv2. > > But maybe that is not needed or would be the wrong direction, so we wanted > to discuss first. > > > > Best regards, > > Felix Mößbauer > > Siemens AG > > Sebastian Best regards, Felix
