On Tue, 1 Nov 2022, at 09:31, Hyeonggon Yoo wrote: > On Tue, Nov 01, 2022 at 09:20:21AM +0000, John Thomson wrote: >> On Tue, 1 Nov 2022, at 07:57, Feng Tang wrote: >> > Hi Thomson, >> > >> > Thanks for testing! >> > >> > + mips maintainer and mail list. The original report is here >> > >> > https://lore.kernel.org/lkml/becf2ac3-2a90-4f3a-96d9-a70f67c66e4a@xxxxxxxxxxxxxxxx/ >> >> I am guessing my issue comes from __kmem_cache_alloc_lru accessing s->object_size when (kmem_cache) s is NULL? >> If that is the case, this change is not to blame, it only exposes the issue? >> >> I get the following dmesg (note very early NULL kmem_cache) with the below change atop v6.1-rc3: >> >> transfer started ......................................... transfer ok, time=2.02s >> setting up elf image... OK >> jumping to kernel code >> zimage at: 80B842A0 810B4EFC >> >> Uncompressing Linux at load address 80001000 >> >> Copy device tree to address 80B80EE0 >> >> Now, booting the kernel... >> >> [ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #61 SMP Tue Nov 1 18:04:13 AEST 2022 >> [ 0.000000] slub: kmem_cache_alloc called with kmem_cache: 0x0 >> [ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache: 0x0 >> [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3 >> [ 0.000000] printk: bootconsole [early0] enabled >> [ 0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc) >> [ 0.000000] MIPS: machine is MikroTik RouterBOARD 760iGS >> >> normal boot >> >> >> diff --git a/mm/slub.c b/mm/slub.c >> index 157527d7101b..10fcdf2520d2 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -3410,7 +3410,13 @@ static __always_inline >> void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, >> gfp_t gfpflags) >> { >> - void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size); >> + void *ret; >> + if (IS_ERR_OR_NULL(s)) { >> + pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s); >> + ret = slab_alloc(s, lru, gfpflags, _RET_IP_, 0); >> + } else { >> + ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size); >> + } >> >> trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE); >> >> @@ -3419,6 +3425,8 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, >> >> void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) >> { >> + if (IS_ERR_OR_NULL(s)) >> + pr_warn("slub: kmem_cache_alloc called with kmem_cache: %pSR\n", s); >> return __kmem_cache_alloc_lru(s, NULL, gfpflags); >> } >> EXPORT_SYMBOL(kmem_cache_alloc); >> @@ -3426,6 +3434,8 @@ EXPORT_SYMBOL(kmem_cache_alloc); >> void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, >> gfp_t gfpflags) >> { >> + if (IS_ERR_OR_NULL(s)) >> + pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s); >> return __kmem_cache_alloc_lru(s, lru, gfpflags); >> } >> EXPORT_SYMBOL(kmem_cache_alloc_lru); >> >> >> Any hints on where kmem_cache_alloc would be being called from this early? >> I will start looking from /init/main.c around pr_notice("%s", linux_banner); > > Great. Would you try calling dump_stack(); when we observed s == NULL? > That would give more information about who passed s == NULL to these > functions. > With the dump_stack() in place: Now, booting the kernel... [ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #62 SMP Tue Nov 1 19:49:52 AEST 2022 [ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache ptr: 0x0 [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.1.0-rc3+ #62 [ 0.000000] Stack : 810fff78 80084d98 80889d00 00000004 00000000 00000000 80889d5c 80c90000 [ 0.000000] 80920000 807bd380 8089d368 80923bd3 00000000 00000001 80889d08 00000000 [ 0.000000] 00000000 00000000 807bd380 8084bd51 00000002 00000002 00000001 6d6f4320 [ 0.000000] 00000000 80c97ce9 80c97d14 fffffffc 807bd380 00000000 00000003 00000dc0 [ 0.000000] 00000000 a0000000 80910000 8110a0b4 00000000 00000020 80010000 80010000 [ 0.000000] ... [ 0.000000] Call Trace: [ 0.000000] [<80008260>] show_stack+0x28/0xf0 [ 0.000000] [<8070cdc0>] dump_stack_lvl+0x60/0x80 [ 0.000000] [<801c1428>] kmem_cache_alloc+0x5c0/0x740 [ 0.000000] [<8092856c>] prom_soc_init+0x1fc/0x2b4 [ 0.000000] [<80928060>] prom_init+0x44/0xf0 [ 0.000000] [<80929214>] setup_arch+0x4c/0x6a8 [ 0.000000] [<809257e0>] start_kernel+0x88/0x7c0 [ 0.000000] [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3 Now, booting the kernel... [ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #62 SMP Tue Nov 1 19:49:52 AEST 2022 [ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache ptr: 0x0 [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.1.0-rc3+ #62 [ 0.000000] Stack : 810fff78 80084d98 80889d00 00000004 00000000 00000000 80889d5c 80c90000 [ 0.000000] 80920000 807bd380 8089d368 80923bd3 00000000 00000001 80889d08 00000000 [ 0.000000] 00000000 00000000 807bd380 8084bd51 00000002 00000002 00000001 6d6f4320 [ 0.000000] 00000000 80c97ce9 80c97d14 fffffffc 807bd380 00000000 00000003 00000dc0 [ 0.000000] 00000000 a0000000 80910000 8110a0b4 00000000 00000020 80010000 80010000 [ 0.000000] ... [ 0.000000] Call Trace: [ 0.000000] show_stack (/mnt/pool_ssd/code/linux/linux-stable-mt7621/./arch/mips/include/asm/stacktrace.h:43 /mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/kernel/traps.c:223) [ 0.000000] dump_stack_lvl (/mnt/pool_ssd/code/linux/linux-stable-mt7621/lib/dump_stack.c:107 (discriminator 1)) [ 0.000000] kmem_cache_alloc (/mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3318 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3406 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3418 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3430) [ 0.000000] prom_soc_init (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/mt7621.c:106 /mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/mt7621.c:177) [ 0.000000] prom_init (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/prom.c:64) [ 0.000000] setup_arch (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/kernel/setup.c:786) [ 0.000000] start_kernel (/mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:279 /mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:477 /mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:960) [ 0.000000] [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3 I have not found it yet. Cheers, -- John Thomson
