hi all, updating the RT kernel from 4.14 to 4.19, we're experiencing a lockup of of RT threads: "Worker 1" and "Worker 2" are my RT threads, which run at high SCHED_FIFO priorities. they are both synchronised via the same futex. wildly speculating, we might run into a race condition between `futex_wake` while `try_to_wake_up -> futex_wait` is this a known issue (it seems that rt48 has some RT related changes)? or is there a good way to trace this down? cyclictest doesn't seem to trigger this situation, with our real-world load, it takes hours to trigger it. thanks a lot, tim --- rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: rcu: Tasks blocked on level-0 rcu_node (CPUs 0-3): P693 P692 rcu: (detected by 0, t=21002 jiffies, g=463681, q=2032) Worker 2 D 0 693 1 0x00000000 Call Trace: ? __schedule+0x2cf/0x770 schedule+0x43/0xe0 rt_spin_lock_slowlock_locked+0x11b/0x2c0 rt_spin_lock_slowlock+0x41/0x60 futex_wake+0xa2/0x150 do_futex+0x942/0xd90 ? trace_hardirqs_on_thunk+0x1a/0x1c ? do_preempt_schedule_irq+0x17/0x17 ? do_futex+0xd90/0xd90 __x64_sys_futex+0x136/0x14f ? do_syscall_64+0x1e/0x3f0 do_syscall_64+0x5d/0x3f0 ? trace_hardirqs_off_thunk+0x1a/0x1c entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x7f9f9e9a0311 Code: ff ff ff 7f 74 4b 48 8d 50 01 f0 48 0f b1 17 75 ee 48 c1 e8 20 74 1b 40 80 f6 81 45 31 d2 ba 01 00 00 00 b8 ca 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 83 c0 16 83 e0 f7 74 RSP: 002b:00007f9f7affca28 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9f9e9a0311 RDX: 0000000000000001 RSI: 0000000000000081 RDI: 000055b8f8c20a88 RBP: 0000000000000001 R08: 000055b9072f2820 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000055b8f8c20a88 R13: 00007f9f7affca97 R14: 00007f9f7affca98 R15: 00007f9f7affcaa0 Worker 1 R running task 0 692 1 0x00000000 Call Trace: ? __schedule+0x2cf/0x770 ? ___preempt_schedule+0x16/0x18 preempt_schedule_common+0x32/0xc0 ___preempt_schedule+0x16/0x18 ? migrate_disable+0x38/0xc0 ? migrate_disable+0x38/0xc0 migrate_disable+0x6c/0xc0 get_futex_value_locked+0x14/0x60 futex_wait_setup+0x80/0x100 ? futex_wait_queue_me+0xbf/0x120 futex_wait+0xb9/0x210 ? try_to_wake_up+0x230/0x6c0 do_futex+0x134/0xd90 ? trace_hardirqs_on_thunk+0x1a/0x1c ? do_preempt_schedule_irq+0x17/0x17 ? trace_hardirqs_on_thunk+0x1a/0x1c ? trace_hardirqs_on_caller+0x32/0x130 ? trace_hardirqs_on_thunk+0x1a/0x1c ? do_preempt_schedule_irq+0x17/0x17 __x64_sys_futex+0x136/0x14f ? do_syscall_64+0x1e/0x3f0 do_syscall_64+0x5d/0x3f0 ? trace_hardirqs_off_thunk+0x1a/0x1c entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x7f9f9e99fd54 Code: e8 41 0b 00 00 b9 ca 00 00 00 41 89 c4 89 de 41 b9 ff ff ff ff 45 31 c0 81 f6 89 01 00 00 45 31 d2 31 d2 48 89 ef 89 c8 0f 05 <48> 3d 00 f0 ff ff 77 14 44 89 e7 e8 6c 0b 00 00 5b 31 c0 5d 41 5c RSP: 002b:00007f9f7b7fda80 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9f9e99fd54 RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055b8f8c20a88 RBP: 000055b8f8c20a88 R08: 0000000000000000 R09: 00000000ffffffff R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: fffffffeffffffff R14: 000055b8f8c20a88 R15: 00007f9f7b7fe700
