On Sat, 12 Mar 2022 17:39:40 -0300 Marcelo Tosatti wrote: > On Fri, Mar 11, 2022 at 09:35:49AM +0100, Sebastian Andrzej Siewior wrote: > > + sched division > > > > On 2022-03-10 18:23:26 [-0800], Andrew Morton wrote: > > > On Thu, 10 Mar 2022 10:22:12 -0300 Marcelo Tosatti <mtosatti@xxxxxxxxxx> wrote: > > > > On systems that run FIFO:1 applications that busy loop, > > > > any SCHED_OTHER task that attempts to execute > > > > on such a CPU (such as work threads) will not > > > > be scheduled, which leads to system hangs. > > … > > > > > > Permitting a realtime thread to hang the entire system warrants a > > > -stable backport, I think. That's just rude. > > > > I'm not sure if someone is not willingly breaking the system. Based on > > my experience, a thread with an elevated priority (that FIFO, RR or DL) > > should not hog the CPU. A normal user (!root && !CAP_SYS_NICE) can't > > increase the priority of the task. > > To avoid a system hangup there is sched_rt_runtime_us which ensures that > > all RT threads are throttled once the RT class exceed a certain amount > > of runtime. This has been relaxed a little on systems with more CPUs so > > that the RT runtime can be shared but this sharing (RT_RUNTIME_SHARE) > > has been disabled by default a while ago. That safe switch > > (sched_rt_runtime_us) can be disabled and is usually disabled on RT > > system since the RT tasks usually run longer especially in corner cases. > > Sebastian, > > Certain classes of applications appear to benefit from very low latency > (or rather very low, to zero, task interruption length). For example, > 5G RAN processing might require maximum interruption of < 10us. > > IIRC the resolution of sched_rt_runtime_us being too high led to the > creation of stalld: > > https://github.com/bristot/stalld > > The stalld program (which stands for 'stall daemon') is a mechanism to > prevent the starvation of operating system threads in a Linux system. > The premise is to start up on a housekeeping cpu (one that is not used > for real-application purposes) and to periodically monitor the state of > each thread in the system, looking for a thread that has been on a run > queue (i.e. ready to run) for a specifed length of time without being > run. This condition is usually hit when the thread is on the same cpu > as a high-priority cpu-intensive task and therefore is being given no > opportunity to run. > > When a thread is judged to be starving, stalld changes that thread to > use the SCHED_DEADLINE policy and gives the thread a small slice of time > for that cpu (specified on the command line). The thread then runs and > when that timeslice is used, the thread is then returned to its original > scheduling policy and stalld then continues to monitor thread states. > > --- > > Configuring 10us per second for time slice of lower priority tasks > that stalld schedules, you'd still get a higher interruption than > 10us to the "RT" high priority application. > > So our impression seems to be that for such low latency requirements its > better to avoid any lower priority work at all on the isolated CPUs. Wonder if it is not a part of isolation to drain the lru cache. > > Yes, there are still cases where work queues are required to do certain > things... we have been trying to reduce such cases. If workqueues are still required on the isolated CPUs then how to walk around the system hang mentioned at the begining of your commit log? Hillf > > > People often isolate CPUs and have busy-loop tasks running with > > SCHED_OTHER given that there is nothing else going on there will be no > > preemption. In this case, the worker would preempt the task. > > In this scenario I _can_ understand that one wants to avoid that > > preemption and try things differently like this patch suggests. We can > > even offload RCU thread from isolated CPUs. > > But I wouldn't say this requires a backport because there is way for a > > RT thread, that claims 100% of the CPU, to break the system. > > On RHEL-8 we had patches (from -RT tree) to avoid workqueues for > flushing memory. So for RHEL-8 -> RHEL-9 the lack of the patch above > is a regression (and the usage of workqueues and not performing > the flush remotely). > > But one might still see the pagevecs not being empty at > __lru_add_drain_all: > > if (pagevec_count(&per_cpu(lru_pvecs.lru_add, cpu)) || > data_race(pagevec_count(&per_cpu(lru_rotate.pvec, cpu))) || > pagevec_count(&per_cpu(lru_pvecs.lru_deactivate_file, cpu)) || > pagevec_count(&per_cpu(lru_pvecs.lru_deactivate, cpu)) || > pagevec_count(&per_cpu(lru_pvecs.lru_lazyfree, cpu)) || > need_activate_page_drain(cpu) || > has_bh_in_lru(cpu, NULL)) { > INIT_WORK(work, lru_add_drain_per_cpu); > queue_work_on(cpu, mm_percpu_wq, work); > __cpumask_set_cpu(cpu, &has_work); > } > > Which needs fixing for a complete solution.
