Hello!
When running rcutorture scenario TREE03 on both next-20240820 and
next-20240821, I see failures like this about half a second into the run
("2024.08.21-11.24.13" on my laptop in case I overtrimmed):
------------------------------------------------------------------------
WARNING: CPU: 4 PID: 42 at kernel/sched/rt.c:1405 dequeue_rt_stack+0x246/0x290
Modules linked in:
CPU: 4 UID: 0 PID: 42 Comm: cpuhp/4 Not tainted 6.11.0-rc1-00048-gaef6987d8954 #152
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
RIP: 0010:dequeue_rt_stack+0x246/0x290
Code: f6 66 89 73 24 83 fa 63 0f 86 ad fe ff ff 90 0f 0b 90 e9 a4 fe ff ff 44 89 ee 48 89 df f7 de e8 50 22 ff ff e9 51 ff ff ff 90 <0f> 0b 90 e9 3a fe ff ff 90 0f 0b 90 e9 ef fd ff ff 8b 14 25 94 fe
RSP: 0000:ffffbc07801dfc18 EFLAGS: 00010046
RAX: ffff9ab05f22c200 RBX: ffff9ab04182e8c0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9ab05f22c200 RDI: ffff9ab04182e8c0
RBP: 000000000002c200 R08: ffffbc07801dfcf8 R09: ffff9ab04182efb4
R10: 0000000000000001 R11: 00000000ffffffff R12: ffff9ab04182e8c0
R13: 0000000000000001 R14: 000000000002c200 R15: 0000000000000008
FS: 0000000000000000(0000) GS:ffff9ab05f300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000011e2e000 CR4: 00000000000006f0
Call Trace:
<TASK>
? __warn+0x7e/0x120
? dequeue_rt_stack+0x246/0x290
? report_bug+0x18e/0x1a0
? handle_bug+0x3d/0x70
? exc_invalid_op+0x18/0x70
? asm_exc_invalid_op+0x1a/0x20
? dequeue_rt_stack+0x246/0x290
dequeue_task_rt+0x68/0x160
move_queued_task.constprop.0+0x62/0xf0
affine_move_task+0x4a3/0x4d0
? affine_move_task+0x229/0x4d0
__set_cpus_allowed_ptr+0x4e/0xa0
set_cpus_allowed_ptr+0x36/0x60
rcutree_affinity_setting+0x16a/0x1d0
? __pfx_rcutree_online_cpu+0x10/0x10
rcutree_online_cpu+0x55/0x60
cpuhp_invoke_callback+0x2cd/0x470
------------------------------------------------------------------------
My reproducer on the two-socket 40-core 80-HW-thread systems is:
tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "50*TREE03" --trust-make
This happens maybe half the time on a two-socket x86 system, and rather
less frequently on my 8-core (16 hardware threads) x86 laptop. (I cheat
and use kvm-remote.sh on 10 two-socket x86 systems to speed things up
a bit.) But bisection is uncharacteristically easy (once I got another
next-20240820 bug out of the way), and converges here:
2e0199df252a ("sched/fair: Prepare exit/cleanup paths for delayed_dequeue")
The preceding commit is very reliable.
Only instead of (or maybe as well as?) introducing the dequeue_rt_stack()
bug, the 2e0199df252a commit introduced a build bug:
------------------------------------------------------------------------
In file included from kernel/sched/fair.c:54:
kernel/sched/fair.c: In function ‘switched_from_fair’:
kernel/sched/sched.h:2154:58: error: ‘__SCHED_FEAT_DELAY_ZERO’ undeclared (first use in this function); did you mean ‘__SCHED_FEAT_LATENCY_WARN’?
2154 | #define sched_feat(x) !!(sysctl_sched_features & (1UL << __SCHED_FEAT_##x))
| ^~~~~~~~~~~~~
kernel/sched/fair.c:12878:21: note: in expansion of macro ‘sched_feat’
12878 | if (sched_feat(DELAY_ZERO) && p->se.vlag > 0)
| ^~~~~~~~~~
kernel/sched/sched.h:2154:58: note: each undeclared identifier is reported only once for each function it appears in
2154 | #define sched_feat(x) !!(sysctl_sched_features & (1UL << __SCHED_FEAT_##x))
| ^~~~~~~~~~~~~
kernel/sched/fair.c:12878:21: note: in expansion of macro ‘sched_feat’
12878 | if (sched_feat(DELAY_ZERO) && p->se.vlag > 0)
| ^~~~~~~~~~
------------------------------------------------------------------------
This build bug continues through this commit:
152e11f6df293 ("sched/fair: Implement delayed dequeue")
By which time it is also accompanied by this (admittedly trivial) warning:
------------------------------------------------------------------------
kernel/sched/fair.c: In function ‘requeue_delayed_entity’:
kernel/sched/fair.c:6818:24: error: unused variable ‘cfs_rq’ [-Werror=unused-variable]
6818 | struct cfs_rq *cfs_rq = cfs_rq_of(se);
| ^~~~~~
------------------------------------------------------------------------
The commit following this one is:
54a58a7877916 ("sched/fair: Implement DELAY_ZERO")
This builds cleanly, but suffers from the dequeue_rt_stack() bug whose
splat is shown above.
Thoughts?
Thanx, Paul
