On Thu, Nov 01, 2018 at 10:01:05AM +0000, John Garry wrote:
> On 31/10/2018 20:46, Peter Zijlstra wrote:
> > On Tue, Oct 30, 2018 at 03:35:35PM +0000, John Garry wrote:
> > > [ 7.154740] ERROR: Node-distance not symmetric
> > > [ 7.154740]
> > > [ 7.160724] 10 15 20 25
> > > [ 7.163456] 15 10 25 30
> > > [ 7.166190] 20 25 10 15
> > > [ 7.168921] 10 10 15 10
> > > [ 7.171655]
> >
> > But I'm not getting the rest of those errors with my 'reproducer':
> >
> > kvm -smp 4 -m 4G -display none -monitor null -serial stdio -kernel defconfig-build/arch/x86/boot/bzImage -append "sched_debug debug ignore_loglevel earlyprintk=serial,ttyS0,115200,keep numa=fake=4:10,15,20,25,15,10,25,30,20,25,10,15,10,10,15,10,0"
> >
> > [ 0.828331] ERROR: Node-distance not symmetric
> > [ 0.828331]
> > [ 0.829081] 10 15 20 25
> > [ 0.830079] 15 10 25 30
> > [ 0.831079] 20 25 10 15
> > [ 0.832079] 10 10 15 10
> > [ 0.833079]
> > [ 0.834373] CPU0 attaching sched-domain(s):
> > [ 0.835082] domain-0: span=0-3 level=DIE
> > [ 0.836079] groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }
> > [ 0.837082] CPU1 attaching sched-domain(s):
> > [ 0.838081] domain-0: span=0-3 level=DIE
> > [ 0.839079] groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 0:{ span=0 }
> > [ 0.840082] CPU2 attaching sched-domain(s):
> > [ 0.841080] domain-0: span=0-3 level=DIE
> > [ 0.842079] groups: 2:{ span=2 }, 3:{ span=3 }, 0:{ span=0 }, 1:{ span=1 }
> > [ 0.843094] ------------[ cut here ]------------
> > [ 0.844076] kernel BUG at ../mm/slub.c:3901!
> > [ 0.844083] invalid opcode: 0000 [#1] SMP PTI
> > [ 0.845076] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.0-rc8+ #305
> > [ 0.845076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
> > [ 0.845076] RIP: 0010:kfree+0x113/0x160
> > [ 0.845076] Code: 18 48 89 da 4c 89 e6 e8 db 01 c5 00 48 8b 45 00 48 85 c0 75 e4 e9 0e ff ff ff 49 8b 02 f6 c4 80 75 0a 49 8b 42 08 a8 01 75 02 <0f> 0b 49 8b 02 31 f6 f6 c4 80 74 05 41 0f b6 72 51 5b 5d 41 5c 4c
> > [ 0.845076] RSP: 0000:ffffabc080633dc8 EFLAGS: 00010246
> > [ 0.845076] RAX: ffff9f973fff8da0 RBX: ffff9f970000001e RCX: 00000000000000f9
> > [ 0.845076] RDX: 0000000000000000 RSI: ffff9f963ea23c80 RDI: 0000606980000000
> > [ 0.845076] RBP: 0000000000020ac0 R08: 0000000000023c80 R09: ffffffff9f8a10db
> > [ 0.845076] R10: fffff17204000000 R11: 0000000000000001 R12: ffffffff9f8a113d
> > [ 0.845076] R13: 0000000000000003 R14: ffffffffa0ab4820 R15: ffff9f973e5bde00
> > [ 0.845076] FS: 0000000000000000(0000) GS:ffff9f963ea00000(0000) knlGS:0000000000000000
> > [ 0.845076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 0.845076] CR2: 00000000ffffffff CR3: 000000008ea0a000 CR4: 00000000000006f0
> > [ 0.845076] Call Trace:
> > [ 0.845076] destroy_sched_domain+0x3d/0x50
> > [ 0.845076] cpu_attach_domain+0x378/0x680
> > [ 0.845076] ? update_group_capacity+0x20/0x2c0
> > [ 0.845076] build_sched_domains+0xde9/0xed0
> > [ 0.845076] ? set_debug_rodata+0xc/0xc
> > [ 0.845076] sched_init_domains+0x80/0x90
> > [ 0.845076] sched_init_smp+0x1d/0x63
> > [ 0.845076] kernel_init_freeable+0x101/0x23f
> > [ 0.845076] ? rest_init+0xb0/0xb0
> > [ 0.845076] kernel_init+0x5/0x100
> > [ 0.845076] ret_from_fork+0x35/0x40
> >
> > I'll work on that crash though..
>
> The actual crash callchain seems to be
> destroy_sched_domain()->free_sched_groups():
>
> static void free_sched_groups(struct sched_group *sg, int free_sgc)
> {
> ...
> do {
> tmp = sg->next;
>
> if (free_sgc && atomic_dec_and_test(&sg->sgc->ref))***
> kfree(sg->sgc);
>
> ...
> }
>
> *** crash occurs when free_sgc is non-zero and sg->sgc is NULL
Yeah, turns out to be random memory corruption; I've had the crash in a
number of weird places; also GCC version dependent.
KASAN is awesome and pinpointed the problem though.
> And, as I mentioned earlier, I bisected this problem to 58d5af59d55b.
You mean:
051f3ca02e46 ("sched/topology: Introduce NUMA identity node sched domain")
right? and yes indeed! The below fixes my reproducer:
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 9d74371e4aad..039578429c25 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1337,7 +1348,7 @@ void sched_init_numa(void)
int level = 0;
int i, j, k;
- sched_domains_numa_distance = kzalloc(sizeof(int) * nr_node_ids, GFP_KERNEL);
+ sched_domains_numa_distance = kzalloc(sizeof(int) * (nr_node_ids + 1), GFP_KERNEL);
if (!sched_domains_numa_distance)
return;
