On Fri, Nov 09, 2018 at 11:35:31AM +0100, Henrik Austad wrote: > On Fri, Nov 09, 2018 at 11:07:28AM +0100, Henrik Austad wrote: > > From: Henrik Austad <haustad@xxxxxxxxx> > > > > Short story: > > Sorry for the spam, it looks like I was not very specific in /which/ > version I targeted this to, as well as not providing a full Cc-list for the > cover-letter. Gentle prod. I realize this was sent out just before plumbers and that people had pretty packed agendas, so a small nudge to gain a spot closer to the top of the inbox :) This series has now been running on an arm64 system for 9 days without any issues and pi_stress showed a dramatic improvement from ~30 seconds and up to several ours (it finally deadlocked at 3.9e9 inversions). I'd greatly appreciate if someone could give the list of patches a quick glance to verify that I got all the required patches and then if it could be added to 4.4.y. Thanks! -Henrik > The series is targeted at stable v4.4.162. > > Expanding Cc-list to those missing from the first attempt. > > -Henrik > > > The following patches are needed on a 4.4 kernel to avoid > > Oops in the scheduler when a sched_rr and sched_deadline task contends > > on the same futex (with PI). > > > > Longer story: > > > > On one of our arm64 systems, we occasionally crash with an Oops in the > > scheduler with the following backtrace. > > > > [<ffffffc0000ee398>] enqueue_task_dl+0x1f0/0x420 > > [<ffffffc0000d0f14>] activate_task+0x7c/0x90 > > [<ffffffc0000edbdc>] push_dl_task+0x164/0x1c8 > > [<ffffffc0000edc60>] push_dl_tasks+0x20/0x30 > > [<ffffffc0000cc00c>] __balance_callback+0x44/0x68 > > [<ffffffc000d2c018>] __schedule+0x6f0/0x728 > > [<ffffffc000d2c278>] schedule+0x78/0x98 > > [<ffffffc000d2e76c>] __rt_mutex_slowlock+0x9c/0x108 > > [<ffffffc000d2e9d0>] rt_mutex_slowlock+0xd8/0x198 > > [<ffffffc0000f7f28>] rt_mutex_timed_futex_lock+0x30/0x40 > > [<ffffffc00012c1a8>] futex_lock_pi+0x200/0x3b0 > > [<ffffffc00012cf84>] do_futex+0x1c4/0x550 > > [<ffffffc00012d92c>] compat_SyS_futex+0x10c/0x138 > > [<ffffffc00008504c>] __sys_trace_return+0x0/0x4 > > > > This seems to be the same bug Xuneli Pang triggered and fixed in > > e96a7705e7d3 "sched/rtmutex/deadline: Fix a PI crash for deadline > > tasks". As noted by Peter Zijlstra in the previous attempt, this fix > > requires a few other patches, most notably the FUTEX_UNLOCK_PI series > > [1] > > > > Testing this on a dual-core VM I have not been able to reproduce the > > same crash, but pi_stress (part of the rt-test suite) reveals that > > vanilla 4.4.162 behaves rather badly with a mix of deadline and > > sched_(rr|fifo) tasks: > > > > time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000 > > Starting PI Stress Test > > Number of thread groups: 1 > > Duration of test run: infinite > > Number of inversions per group: unlimited > > Admin thread SCHED_RR priority 4 > > 1 groups of 3 threads will be created > > High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000 > > Med thread SCHED_RR priority 2 > > Low thread SCHED_RR priority 1 > > Current Inversions: 141627 > > WATCHDOG triggered: group 0 is deadlocked! > > reporter stopping due to watchdog event > > Stopping test > > Terminated > > > > real 0m26.291s > > user 0m0.148s > > sys 0m18.819s > > > > With this series applied, the test ran for ~4.5 hours and again for 129 > > minutes (when I remembered to time it) before crashing: > > > > time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000 > > Starting PI Stress Test > > Number of thread groups: 1 > > Duration of test run: infinite > > Number of inversions per group: unlimited > > Admin thread SCHED_RR priority 4 > > 1 groups of 3 threads will be created > > High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000 > > Med thread SCHED_RR priority 2 > > Low thread SCHED_RR priority 1 > > Current Inversions: 51985223 > > WATCHDOG triggered: group 0 is deadlocked! > > reporter stopping due to watchdog event > > Stopping test > > Terminated > > > > real 129m38.807s > > user 0m59.084s > > sys 109m53.666s > > > > > > So clearly not perfect, but a *lot* better. > > > > The same series on our vendor-4.4 kernel moves pi_stress up from ~30 > > seconds before deadlock up to the same level as the VM (the test is > > still going as of this writing). > > > > I suspect other users of 4.4 would benefit from having these patches > > backported, so tag them for stable. I assume 4.9 and 4.14 could benefit > > as well, but I have not had time to look into those. > > > > 1) https://www.mail-archive.com/linux-kernel@xxxxxxxxxxxxxxx/msg1359667.html > > > > Peter Zijlstra (13): > > futex: Cleanup variable names for futex_top_waiter() > > futex: Use smp_store_release() in mark_wake_futex() > > futex: Remove rt_mutex_deadlock_account_*() > > futex,rt_mutex: Provide futex specific rt_mutex API > > futex: Change locking rules > > futex: Cleanup refcounting > > futex: Rework inconsistent rt_mutex/futex_q state > > futex: Pull rt_mutex_futex_unlock() out from under hb->lock > > futex,rt_mutex: Introduce rt_mutex_init_waiter() > > futex,rt_mutex: Restructure rt_mutex_finish_proxy_lock() > > futex: Rework futex_lock_pi() to use rt_mutex_*_proxy_lock() > > futex: Futex_unlock_pi() determinism > > futex: Drop hb->lock before enqueueing on the rtmutex > > > > Thomas Gleixner (2): > > rtmutex: Make wait_lock irq safe > > futex: Rename free_pi_state() to put_pi_state() > > > > Xunlei Pang (2): > > rtmutex: Deboost before waking up the top waiter > > sched/rtmutex/deadline: Fix a PI crash for deadline tasks > > > > include/linux/init_task.h | 1 + > > include/linux/sched.h | 2 + > > include/linux/sched/rt.h | 1 + > > kernel/fork.c | 1 + > > kernel/futex.c | 532 ++++++++++++++++++++++++++-------------- > > kernel/locking/rtmutex-debug.c | 9 - > > kernel/locking/rtmutex-debug.h | 3 - > > kernel/locking/rtmutex.c | 406 ++++++++++++++++++------------ > > kernel/locking/rtmutex.h | 2 - > > kernel/locking/rtmutex_common.h | 24 +- > > kernel/sched/core.c | 2 + > > 11 files changed, 620 insertions(+), 363 deletions(-) > > > > -- > > 2.7.4 > > > > -- > Henrik Austad -- Henrik Austad
Attachment:
signature.asc
Description: PGP signature
