Dear subscribers, we are currently evaluating how to rework realtime tuning to use cgroup-v2 cpusets instead of the isolcpus kernel parameter. Our use-case are realtime applications with rt and non-rt threads. Hereby, the non-rt thread might create additional non-rt threads: Example (RT CPU=1, 4 CPUs): - Non-RT Thread (A) with default affinity 0xD (1101b) - RT Thread (B) with Affinity 0x2 (0010b, via set_affinity) When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly works: Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a default affinity of 0xD. By that, independent of the threads' priorities, we can ensure that nothing is scheduled on our RT cpu (except from kernel threads, etc...). During this journey, we discovered the following: Using cgroup-v2 cpusets and isolcpus together seems to be incompatible: When activating the cpuset controller on a cgroup (for the first time), all default CPU affinities are reset. By that, also the default affinity is set to 0xFFFF..., while with isolcpus we expect it to be (0xFFFF - isolcpus). This breaks the example from above, as now the non-RT thread can also be scheduled on the RT CPU. When only using cgroup-v2, we can isolate our RT process by placing it in a cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups. However, we do not know of a strategy to set a default affinity: Given the example above, we have no way to ensure that newly created threads are born with an affinity of just 0x2 (without changing the application). Finally, isolcpus itself is deprecated since kernel 5.4. Questions: 1. What is the best strategy to "isolcpus" similar semantics with cgroups-v2? 2. Is there a way to specify the default affinity (within a cgroup) We are currently at a point where we would write patches to add a default affinity feature to cpusets of cgroupv2. But maybe that is not needed or would be the wrong direction, so we wanted to discuss first. Best regards, Felix Mößbauer Siemens AG
