Thanks Julia. You've been of great help. Quick question. How do you know the wakeup occurs at timestamp 12321.992480? At that timestamp the only thing we see is: <idle>-0 [003] .n..1.. 12321.992480: rcu_idle_exit <-cpu_startup_entry <idle>-0 [003] dn..1.. 12321.992480: rcu_eqs_exit_common.isra.46 <-rcu_idle_exit <idle>-0 [003] .n..1.. 12321.992480: arch_cpu_idle_exit <-cpu_startup_entry <idle>-0 [003] .n..1.. 12321.992480: atomic_notifier_call_chain <-arch_cpu_idle_exit And we also see this: tuators_manager-1512 [003] ....1.. 12321.992495: sys_futex(uaddr: 7f41cc000020, op: 81, val: 1, utime: 7f41e4a60f19, uaddr2: 0, val3: 302e332e312e31) Can you explain us how is it related to the same futex, please? We see this call repeatedly across all the trace. We'll try tracing the other threads to pick who issues the FUTEX_WAKE. Kind regards, Jordan. On 25 May 2018 at 17:02, Julia Cartwright <julia@xxxxxx> wrote: > On Fri, May 25, 2018 at 03:38:47PM +0200, Jordan Palacios wrote: >> Hello, >> >> We managed to trace one of the failing cycles. The trace is here: >> >> https://pastebin.com/YJBrSQpJ >> >> It seems our application is relinquishing the cpu (line 411) due to a >> sys_futex call (line 350). We still don't understand why though. We >> are not very familiar with all the kernel functions. > > You'll see a futex system call whenever there is contention on a > userspace mutex. In this case, your thread is issuing a FUTEX_WAIT, > which is the futex op used to implement a contended non-PI lock > operation. > > An RT task blocking on a non-PI mutex is game over w.r.t. expecting any > bound execution time guarantees. > > In other words: the traces show that this is a userspace problem, not a > kernel problem. Solving this will require you to inspect your > application's locking. > > It may be helpful for you, in this effort, to identify the other thread > which eventually issues the FUTEX_WAKE (used for non-PI unlock > operation); the trace you linked only includes traces for CPU3, the > waker is on another CPU. The remote wakeup occurs at timestamp > 12321.992480. > > Julia -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
