On Fri, Nov 09, 2018 at 11:07:28AM +0100, Henrik Austad wrote: > From: Henrik Austad <haustad@xxxxxxxxx> > > Short story: Sorry for the spam, it looks like I was not very specific in /which/ version I targeted this to, as well as not providing a full Cc-list for the cover-letter. The series is targeted at stable v4.4.162. Expanding Cc-list to those missing from the first attempt. -Henrik > The following patches are needed on a 4.4 kernel to avoid > Oops in the scheduler when a sched_rr and sched_deadline task contends > on the same futex (with PI). > > Longer story: > > On one of our arm64 systems, we occasionally crash with an Oops in the > scheduler with the following backtrace. > > [<ffffffc0000ee398>] enqueue_task_dl+0x1f0/0x420 > [<ffffffc0000d0f14>] activate_task+0x7c/0x90 > [<ffffffc0000edbdc>] push_dl_task+0x164/0x1c8 > [<ffffffc0000edc60>] push_dl_tasks+0x20/0x30 > [<ffffffc0000cc00c>] __balance_callback+0x44/0x68 > [<ffffffc000d2c018>] __schedule+0x6f0/0x728 > [<ffffffc000d2c278>] schedule+0x78/0x98 > [<ffffffc000d2e76c>] __rt_mutex_slowlock+0x9c/0x108 > [<ffffffc000d2e9d0>] rt_mutex_slowlock+0xd8/0x198 > [<ffffffc0000f7f28>] rt_mutex_timed_futex_lock+0x30/0x40 > [<ffffffc00012c1a8>] futex_lock_pi+0x200/0x3b0 > [<ffffffc00012cf84>] do_futex+0x1c4/0x550 > [<ffffffc00012d92c>] compat_SyS_futex+0x10c/0x138 > [<ffffffc00008504c>] __sys_trace_return+0x0/0x4 > > This seems to be the same bug Xuneli Pang triggered and fixed in > e96a7705e7d3 "sched/rtmutex/deadline: Fix a PI crash for deadline > tasks". As noted by Peter Zijlstra in the previous attempt, this fix > requires a few other patches, most notably the FUTEX_UNLOCK_PI series > [1] > > Testing this on a dual-core VM I have not been able to reproduce the > same crash, but pi_stress (part of the rt-test suite) reveals that > vanilla 4.4.162 behaves rather badly with a mix of deadline and > sched_(rr|fifo) tasks: > > time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000 > Starting PI Stress Test > Number of thread groups: 1 > Duration of test run: infinite > Number of inversions per group: unlimited > Admin thread SCHED_RR priority 4 > 1 groups of 3 threads will be created > High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000 > Med thread SCHED_RR priority 2 > Low thread SCHED_RR priority 1 > Current Inversions: 141627 > WATCHDOG triggered: group 0 is deadlocked! > reporter stopping due to watchdog event > Stopping test > Terminated > > real 0m26.291s > user 0m0.148s > sys 0m18.819s > > With this series applied, the test ran for ~4.5 hours and again for 129 > minutes (when I remembered to time it) before crashing: > > time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000 > Starting PI Stress Test > Number of thread groups: 1 > Duration of test run: infinite > Number of inversions per group: unlimited > Admin thread SCHED_RR priority 4 > 1 groups of 3 threads will be created > High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000 > Med thread SCHED_RR priority 2 > Low thread SCHED_RR priority 1 > Current Inversions: 51985223 > WATCHDOG triggered: group 0 is deadlocked! > reporter stopping due to watchdog event > Stopping test > Terminated > > real 129m38.807s > user 0m59.084s > sys 109m53.666s > > > So clearly not perfect, but a *lot* better. > > The same series on our vendor-4.4 kernel moves pi_stress up from ~30 > seconds before deadlock up to the same level as the VM (the test is > still going as of this writing). > > I suspect other users of 4.4 would benefit from having these patches > backported, so tag them for stable. I assume 4.9 and 4.14 could benefit > as well, but I have not had time to look into those. > > 1) https://www.mail-archive.com/linux-kernel@xxxxxxxxxxxxxxx/msg1359667.html > > Peter Zijlstra (13): > futex: Cleanup variable names for futex_top_waiter() > futex: Use smp_store_release() in mark_wake_futex() > futex: Remove rt_mutex_deadlock_account_*() > futex,rt_mutex: Provide futex specific rt_mutex API > futex: Change locking rules > futex: Cleanup refcounting > futex: Rework inconsistent rt_mutex/futex_q state > futex: Pull rt_mutex_futex_unlock() out from under hb->lock > futex,rt_mutex: Introduce rt_mutex_init_waiter() > futex,rt_mutex: Restructure rt_mutex_finish_proxy_lock() > futex: Rework futex_lock_pi() to use rt_mutex_*_proxy_lock() > futex: Futex_unlock_pi() determinism > futex: Drop hb->lock before enqueueing on the rtmutex > > Thomas Gleixner (2): > rtmutex: Make wait_lock irq safe > futex: Rename free_pi_state() to put_pi_state() > > Xunlei Pang (2): > rtmutex: Deboost before waking up the top waiter > sched/rtmutex/deadline: Fix a PI crash for deadline tasks > > include/linux/init_task.h | 1 + > include/linux/sched.h | 2 + > include/linux/sched/rt.h | 1 + > kernel/fork.c | 1 + > kernel/futex.c | 532 ++++++++++++++++++++++++++-------------- > kernel/locking/rtmutex-debug.c | 9 - > kernel/locking/rtmutex-debug.h | 3 - > kernel/locking/rtmutex.c | 406 ++++++++++++++++++------------ > kernel/locking/rtmutex.h | 2 - > kernel/locking/rtmutex_common.h | 24 +- > kernel/sched/core.c | 2 + > 11 files changed, 620 insertions(+), 363 deletions(-) > > -- > 2.7.4 > -- Henrik Austad
Attachment:
signature.asc
Description: PGP signature
