Hi, Dave On 12/01/15 at 04:49pm, Dave Anderson wrote: > > ----- Original Message ----- > > On 11/29/15 at 10:52pm, Dave Young wrote: > > [snip] > > > > With the new build it fails at later point, but I noticed I enabled > > CONFIG_DEBUG_INFO_REDUCED=y > > > > A retest without CONFIG_DEBUG_INFO_REDUCED succeed with latest latest > > crash build, but there's some read errors. I'm not sure if they matters > > but "bt" works well in crash. > > > > [snip] > > GNU gdb (GDB) 7.6 > > Copyright (C) 2013 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > > and "show warranty" for details. > > This GDB was configured as "armv7l-unknown-linux-gnueabihf"... > > > > please wait... (gathering kmem slab cache data) > > crash: read error: kernel virtual address: ff7fb414 type: "array cache > > limit" > > > > crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410 > > > > crash: read error: kernel virtual address: ff7fac08 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3540: invalid array_cache pointer: ff7fac04 > > > > crash: read error: kernel virtual address: ff7f7edc type: "array cache > > limit" > > > > crash: kmem_cache: df1e35c0: invalid array_cache pointer: ff7f7ed8 > > > > crash: read error: kernel virtual address: ff7f7cec type: "array cache > > limit" > > > > crash: kmem_cache: df1e3640: invalid array_cache pointer: ff7f7ce8 > > > > crash: read error: kernel virtual address: ff7f7c04 type: "array cache > > limit" > > > > crash: kmem_cache: df1e36c0: invalid array_cache pointer: ff7f7c00 > > > > crash: read error: kernel virtual address: ff7f7894 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3740: invalid array_cache pointer: ff7f7890 > > > > crash: read error: kernel virtual address: ff7f77ac type: "array cache > > limit" > > > > crash: kmem_cache: df1e37c0: invalid array_cache pointer: ff7f77a8 > > > > crash: read error: kernel virtual address: ff7f76c4 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3840: invalid array_cache pointer: ff7f76c0 > > > > crash: read error: kernel virtual address: ff7f74d4 type: "array cache > > limit" > > > > crash: kmem_cache: df1e38c0: invalid array_cache pointer: ff7f74d0 > > > > crash: read error: kernel virtual address: ff7f73ec type: "array cache > > limit" > > > > crash: kmem_cache: df1e3940: invalid array_cache pointer: ff7f73e8 > > > > crash: read error: kernel virtual address: ff7f7304 type: "array cache > > limit" > > > > crash: kmem_cache: df1e39c0: invalid array_cache pointer: ff7f7300 > > > > crash: read error: kernel virtual address: ff7f707c type: "array cache > > limit" > > > > crash: kmem_cache: df1e3a40: invalid array_cache pointer: ff7f7078 > > > > crash: read error: kernel virtual address: ff7f6e8c type: "array cache > > limit" > > > > crash: kmem_cache: df1e3ac0: invalid array_cache pointer: ff7f6e88 > > > > crash: read error: kernel virtual address: ff7f6c9c type: "array cache > > limit" > > > > crash: kmem_cache: df1e3b40: invalid array_cache pointer: ff7f6c98 > > > > crash: read error: kernel virtual address: ff7f6aac type: "array cache > > limit" > > > > crash: kmem_cache: df1e3bc0: invalid array_cache pointer: ff7f6aa8 > > > > crash: read error: kernel virtual address: ff7f68bc type: "array cache > > limit" > > > > crash: kmem_cache: df1e3c40: invalid array_cache pointer: ff7f68b8 > > > > crash: read error: kernel virtual address: ff7f67d4 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3cc0: invalid array_cache pointer: ff7f67d0 > > > > crash: read error: kernel virtual address: ff7f65e4 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3d40: invalid array_cache pointer: ff7f65e0 > > > > crash: read error: kernel virtual address: ff7f63f4 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3dc0: invalid array_cache pointer: ff7f63f0 > > > > crash: read error: kernel virtual address: ff7f6204 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3e40: invalid array_cache pointer: ff7f6200 > > > > crash: read error: kernel virtual address: ff7f6014 type: "array cache > > limit" > > > > crash: kmem_cache: df1e3ec0: invalid array_cache pointer: ff7f6010 > > > > crash: read error: kernel virtual address: ff7f5e0c type: "array cache > > limit" > > > > crash: kmem_cache: df1e3f40: invalid array_cache pointer: ff7f5e08 > > > > crash: read error: kernel virtual address: ff7f5b04 type: "array cache > > limit" > > > > crash: kmem_cache: df017040: invalid array_cache pointer: ff7f5b00 > > > > crash: read error: kernel virtual address: ff7f57e0 type: "array cache > > limit" > > > > crash: kmem_cache: df0170c0: invalid array_cache pointer: ff7f57dc > > > > crash: read error: kernel virtual address: ff7f5300 type: "array cache > > limit" > > > > crash: kmem_cache: df017140: invalid array_cache pointer: ff7f52fc > > > > crash: read error: kernel virtual address: ff7f50ec type: "array cache > > limit" > > > > crash: kmem_cache: df0171c0: invalid array_cache pointer: ff7f50e8 > > > > crash: read error: kernel virtual address: ff7f4004 type: "array cache > > limit" > > > > crash: kmem_cache: df017240: invalid array_cache pointer: ff7f4000 > > WARNING: SLAB: cannot determine how compound pages are linked > > > > KERNEL: vmlinux > > DUMPFILE: /var/crash/127.0.0.1-2015-11-30-13:10:23/vmcore > > CPUS: 1 > > DATE: Mon Nov 30 13:10:14 2015 > > UPTIME: 00:00:38 > > LOAD AVERAGE: 0.27, 0.08, 0.03 > > TASKS: 157 > > NODENAME: localhost > > RELEASE: 4.4.0-rc2+ > > VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015 > > MACHINE: armv7l (unknown Mhz) > > MEMORY: 510 MB > > PANIC: "sysrq: SysRq : Trigger a crash" > > PID: 1368 > > COMMAND: "bash" > > TASK: ddbbeb40 [THREAD_INFO: ddbc4000] > > CPU: 0 > > STATE: TASK_RUNNING (SYSRQ) > > > > crash> bt > > PID: 1368 TASK: ddbbeb40 CPU: 0 COMMAND: "bash" > > #0 [<c021e23c>] (sysrq_handle_crash) from [<c021e5a5>] > > #1 [<c021e5a5>] (__handle_sysrq) from [<c021e969>] > > #2 [<c021e969>] (write_sysrq_trigger) from [<c01067df>] > > #3 [<c01067df>] (proc_reg_write) from [<c00ca5db>] > > #4 [<c00ca5db>] (__vfs_write) from [<c00cac0f>] > > #5 [<c00cac0f>] (vfs_write) from [<c00cb24b>] > > #6 [<c00cb24b>] (sys_write) from [<c000ddc1>] > > crash> > > > > > Can you send me the location of the vmlinux and vmcore files offline? I can > > > try working with them tomorrow with on an x86_64 host with a crash utility > > > built with "make target=ARM". > > > > Will save the files somewhere and let you know.. > > > > Thanks a lot > > Dave > > > > > Dave, > > OK, so the problem is that in CONFIG_SLAB kernels, the array_cache > structure that is pointed to by each kmem_cache.cpu_cache can now > be a vmalloc'd memory location. That may be a new innovation for > 32-bit kernels? I don't know, but in any case, all of these error Hmm, I did not notice the CONFIG_SLAB=y The vmalloc could be introduced by because percpu could use vmalloced area: commit bf0dea23a9c094ae869a88bb694fbe966671bf6d Author: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx> Date: Thu Oct 9 15:26:27 2014 -0700 mm/slab: use percpu allocator for cpu cache > types: > > crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410 > > show array_cache pointers in the vmalloc region. The kmem_cache.cpu_cache > pointer is actually a per_cpu pointer, but on some of the kmem slab caches, > the translation of the per_cpu pointer result in a vmalloc() address instead > of a unity-mapped address. Perhaps there's a point at which the percpu > memory allocations switch from being unity-mapped structures to being > vmalloc()'d addresses? I think you are right though I did not find the point as well. > > In any case, at the point in time when the kmem caches are being > initialized (when the error messages get generated), the 32-bit ARM > arch has not yet set the base address of the vmalloc region in one > if its machine-dependent variables. And so the addresses are > mistakenly translated as if they are unity-mapped. The ARM arch > sets the internal vmalloc base address variable at a later point > in time, but it can be done earlier, before the kmem cache initialization > is started. So with this one-line patch, the dumpfile comes up cleanly: > > --- arm.c.orig 2015-12-01 16:36:11.815775008 -0500 > +++ arm.c 2015-12-01 16:25:44.580800293 -0500 > @@ -1539,6 +1539,7 @@ > static ulong > arm_vmalloc_start(void) > { > + machdep->machspec->vmalloc_start_addr = vt->high_memory; > return vt->high_memory; > } > > $ crash vmlinux vmcore-cp > > crash 7.1.4rc21 > Copyright (C) 2002-2014 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for details. > > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-elf-linux"... > > KERNEL: vmlinux > DUMPFILE: vmcore-cp > CPUS: 1 > DATE: Mon Nov 30 00:10:14 2015 > UPTIME: 00:00:38 > LOAD AVERAGE: 0.27, 0.08, 0.03 > TASKS: 157 > NODENAME: localhost > RELEASE: 4.4.0-rc2+ > VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015 > MACHINE: armv7l (unknown Mhz) > MEMORY: 510 MB > PANIC: "sysrq: SysRq : Trigger a crash" > PID: 1368 > COMMAND: "bash" > TASK: ddbbeb40 [THREAD_INFO: ddbc4000] > CPU: 0 > STATE: TASK_RUNNING (SYSRQ) > > crash> > > I can only test the ELF vmcore because I don't have the 32-bit lzo library > installed for my "make target=ARM" x86 binary. But it won't be any > different from the one above. > > I'll queue the patch for crash-7.1.4. Cool, Thanks a lot Dave -- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility