On Mon, 24 Jul 2023 at 15:50, Alexander Potapenko <glider@xxxxxxxxxx> wrote: > > On Sat, Jul 22, 2023 at 6:37 PM Linus Torvalds > <torvalds@xxxxxxxxxxxxxxxxxxxx> wrote: > > > > [ Removed the stable reviewers, bringing in the kfence people ] > > > > See > > > > https://lore.kernel.org/lkml/CA+G9fYvgy22wiY=c3wLOrCM6o33636abhtEynXhJkqxJh4ca0A@xxxxxxxxxxxxxx/ > > > > for the original report. The warning was introduced in 8f0b36497303 > > ("mm: kfence: fix objcgs vector allocation"), and Google doesn't find > > any other cases of this. > > > > Anybody? > > > > Linus > > > > > > > NOTE: > > > The following kernel warning was noticed while booting qemu-arm64 > > > with these configs enabled on stable rc 6.4.5-rc1. > > > > > > CONFIG_ARM64_64K_PAGES=y > > > CONFIG_KFENCE=y > > Is there a full config somewhere? Please find build details - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/ - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/config - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/vmlinux.xz - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/System.map - https://storage.tuxsuite.com/public/linaro/lkft/builds/2StEPFnEfoD076PRu8fIxjexhnM/Image.gz > > > > This crash is not easily reproducible. > > CONFIG_KFENCE_SAMPLE_INTERVAL=10 > CONFIG_KFENCE_NUM_OBJECTS=2048 > > might improve reproducibility. The above test have following Kconfigs enabled. CONFIG_HAVE_ARCH_KASAN=y CONFIG_HAVE_ARCH_KASAN_SW_TAGS=y CONFIG_HAVE_ARCH_KASAN_HW_TAGS=y CONFIG_HAVE_ARCH_KASAN_VMALLOC=y CONFIG_CC_HAS_KASAN_GENERIC=y CONFIG_CC_HAS_KASAN_SW_TAGS=y CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y # CONFIG_KASAN is not set CONFIG_HAVE_ARCH_KFENCE=y CONFIG_KFENCE=y CONFIG_KFENCE_SAMPLE_INTERVAL=100 CONFIG_KFENCE_NUM_OBJECTS=255 # CONFIG_KFENCE_DEFERRABLE is not set CONFIG_KFENCE_STRESS_TEST_FAULTS=0 > > > > > > > boot logs: > > > -------- > > > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510] > > > [ 0.000000] Linux version 6.4.5-rc1 (tuxmake@tuxmake) > > > (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils > > > for Debian) 2.40) #1 SMP PREEMPT @1689957802 > > > [ 0.000000] random: crng init done > > > [ 0.000000] Machine model: linux,dummy-virt > > > ... > > > [ 0.006821] kfence: initialized - using 33554432 bytes for 255 > > > objects at 0x(____ptrval____)-0x(____ptrval____) > > > ... > > > [ 7.726994] ------------[ cut here ]------------ > > > [ 7.727704] WARNING: CPU: 1 PID: 1 at mm/kfence/core.c:1097 > > > __kfence_free+0x84/0xc8 > > ... > > > [ 7.746478] Call trace: > > > [ 7.746776] __kfence_free+0x84/0xc8 > > > [ 7.747134] __slab_free+0x490/0x508 > > > [ 7.748063] __kmem_cache_free+0x2b4/0x2d0 > > > [ 7.748377] kfree+0x78/0x140 > > > [ 7.748638] single_release+0x40/0x60 > > > [ 7.750664] __fput+0x78/0x260 > > > [ 7.751065] ____fput+0x18/0x30 > > > [ 7.752086] task_work_run+0x80/0xe0 > > > [ 7.753122] do_notify_resume+0x200/0x1398 > > > [ 7.754292] el0_svc+0xec/0x100 > > > [ 7.754573] el0t_64_sync_handler+0xf4/0x120 > > > [ 7.755559] el0t_64_sync+0x190/0x198 > > It would be interesting to see the contents of > /sys/kernel/debug/kfence/objects together with the object address. > Would it be possible to boot the kernel with no_hash_pointers and add > a line printing the object address: > > diff --git a/mm/kfence/core.c b/mm/kfence/core.c > index dad3c0eb70a01..23f27f6cb18cf 100644 > --- a/mm/kfence/core.c > +++ b/mm/kfence/core.c > @@ -1094,7 +1094,10 @@ void __kfence_free(void *addr) > struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); > > #ifdef CONFIG_MEMCG > - KFENCE_WARN_ON(meta->objcg); > + if (meta->objcg) { > + pr_err("ADDR: %px\n", addr); > + KFENCE_WARN_ON(1); > + } > #endif > /* > * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing > > , and then dump /sys/kernel/debug/kfence/objects? > This testing is running on CI loops and however, I will try to reproduce this locally. > Knowing the kfence pool location (the line starting with "kfence: > initialized") and the object address, we can probably understand from > the allocation stack in sysfs, whether the object is supposed to be > deleted. - Naresh
