On Wed, Dec 11, 2019 at 11:08:29PM +0530, Srikar Dronamraju wrote: > A running task can wake-up a per CPU bound kthread on the same CPU. > If the current running task doesn't yield the CPU before the next load > balance operation, the scheduler would detect load imbalance and try to > balance the load. However this load balance would fail as the waiting > task is CPU bound, while the running task cannot be moved by the regular > load balancer. Finally the active load balancer would kick in and move > the task to a different CPU/Core. Moving the task to a different > CPU/core can lead to loss in cache affinity leading to poor performance. > > This is more prone to happen if the current running task is CPU > intensive and the sched_wake_up_granularity is set to larger value. > When the sched_wake_up_granularity was relatively small, it was observed > that the bound thread would complete before the load balancer would have > chosen to move the cache hot task to a different CPU. > > To deal with this situation, the current running task would yield to a > per CPU bound kthread, provided kthread is not CPU intensive. So a question for you here: when does the workqueue worker pre-empt the currently running task? Is it immediately? Or when a time-slice of the currently running task runs out? We don't want queued work immediately pre-empting the task that queued the work - the queued work is *deferred* work that should be run _soon_ but we want the currently running task to finish what it is doing first if possible. i.e. these are not synchronous wakeups, and so we shouldn't schedule kworker threads as though they are sync wakeups. That will affect batch processing effciency and reduce throughput because it will greatly increase the number of unnecessary context switches during IO completion processing.... > /pboffline/hwcct_prg_old/lib/fsperf -t overwrite --noclean -f 5g -b 4k /pboffline > > (With sched_wake_up_granularity set to 15ms) > > Performance counter stats for 'system wide' (5 runs): > event v5.4 v5.4 + patch(v3) > probe:active_load_balance_cpu_stop 1,919 ( +- 2.89% ) 4 ( +- 20.48% ) > sched:sched_waking 441,535 ( +- 0.17% ) 914,630 ( +- 0.18% ) > sched:sched_wakeup 441,533 ( +- 0.17% ) 914,630 ( +- 0.18% ) > sched:sched_wakeup_new 2,436 ( +- 8.08% ) 545 ( +- 4.02% ) > sched:sched_switch 797,007 ( +- 0.26% ) 1,490,261 ( +- 0.10% ) > sched:sched_migrate_task 20,998 ( +- 1.04% ) 2,492 ( +- 11.56% ) As we see here. We've doubled the number of context switches (increased by 700,000) just to avoid 17,000 incorrect load balancer task migrations. That seems like we now make 700,000 incorrect decisions instead of just 20,000. The difference is that the consequence of making these many incorrect pre-emption decisions is vastly less than the consequence of making the wrong migration decision. It seems to me that we should be checking this is_per_cpu_kthread() state for tasks queued on the runqueue during active load balancing, rather than at wakeup time. i.e. in these cases we don't migrate the running task, we just let it run out it's timeslice out and the local per-cpu kthreads then run appropriately. AFAICT this would have the same effect of avoiding unnecessary task migrations in this workload, but without causing a global change to the way workqueue kworkers are scheduled that has the potential to cause regressions in other workqueue intensive workloads.... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
