Re: crash and upstream arm kernel

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



----- Original Message -----
> On 11/29/15 at 10:52pm, Dave Young wrote:
> [snip]
>
> With the new build it fails at later point, but I noticed I enabled
> CONFIG_DEBUG_INFO_REDUCED=y
> 
> A retest without CONFIG_DEBUG_INFO_REDUCED succeed with latest latest
> crash build, but there's some read errors. I'm not sure if they matters
> but "bt" works well in crash.
> 
> [snip]
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "armv7l-unknown-linux-gnueabihf"...
> 
> please wait... (gathering kmem slab cache data)
> crash: read error: kernel virtual address: ff7fb414  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410
> 
> crash: read error: kernel virtual address: ff7fac08  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3540: invalid array_cache pointer: ff7fac04
> 
> crash: read error: kernel virtual address: ff7f7edc  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e35c0: invalid array_cache pointer: ff7f7ed8
> 
> crash: read error: kernel virtual address: ff7f7cec  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3640: invalid array_cache pointer: ff7f7ce8
> 
> crash: read error: kernel virtual address: ff7f7c04  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e36c0: invalid array_cache pointer: ff7f7c00
> 
> crash: read error: kernel virtual address: ff7f7894  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3740: invalid array_cache pointer: ff7f7890
> 
> crash: read error: kernel virtual address: ff7f77ac  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e37c0: invalid array_cache pointer: ff7f77a8
> 
> crash: read error: kernel virtual address: ff7f76c4  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3840: invalid array_cache pointer: ff7f76c0
> 
> crash: read error: kernel virtual address: ff7f74d4  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e38c0: invalid array_cache pointer: ff7f74d0
> 
> crash: read error: kernel virtual address: ff7f73ec  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3940: invalid array_cache pointer: ff7f73e8
> 
> crash: read error: kernel virtual address: ff7f7304  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e39c0: invalid array_cache pointer: ff7f7300
> 
> crash: read error: kernel virtual address: ff7f707c  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3a40: invalid array_cache pointer: ff7f7078
> 
> crash: read error: kernel virtual address: ff7f6e8c  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3ac0: invalid array_cache pointer: ff7f6e88
> 
> crash: read error: kernel virtual address: ff7f6c9c  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3b40: invalid array_cache pointer: ff7f6c98
> 
> crash: read error: kernel virtual address: ff7f6aac  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3bc0: invalid array_cache pointer: ff7f6aa8
> 
> crash: read error: kernel virtual address: ff7f68bc  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3c40: invalid array_cache pointer: ff7f68b8
> 
> crash: read error: kernel virtual address: ff7f67d4  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3cc0: invalid array_cache pointer: ff7f67d0
> 
> crash: read error: kernel virtual address: ff7f65e4  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3d40: invalid array_cache pointer: ff7f65e0
> 
> crash: read error: kernel virtual address: ff7f63f4  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3dc0: invalid array_cache pointer: ff7f63f0
> 
> crash: read error: kernel virtual address: ff7f6204  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3e40: invalid array_cache pointer: ff7f6200
> 
> crash: read error: kernel virtual address: ff7f6014  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3ec0: invalid array_cache pointer: ff7f6010
> 
> crash: read error: kernel virtual address: ff7f5e0c  type: "array cache
> limit"
> 
> crash: kmem_cache: df1e3f40: invalid array_cache pointer: ff7f5e08
> 
> crash: read error: kernel virtual address: ff7f5b04  type: "array cache
> limit"
> 
> crash: kmem_cache: df017040: invalid array_cache pointer: ff7f5b00
> 
> crash: read error: kernel virtual address: ff7f57e0  type: "array cache
> limit"
> 
> crash: kmem_cache: df0170c0: invalid array_cache pointer: ff7f57dc
> 
> crash: read error: kernel virtual address: ff7f5300  type: "array cache
> limit"
> 
> crash: kmem_cache: df017140: invalid array_cache pointer: ff7f52fc
> 
> crash: read error: kernel virtual address: ff7f50ec  type: "array cache
> limit"
> 
> crash: kmem_cache: df0171c0: invalid array_cache pointer: ff7f50e8
> 
> crash: read error: kernel virtual address: ff7f4004  type: "array cache
> limit"
> 
> crash: kmem_cache: df017240: invalid array_cache pointer: ff7f4000
> WARNING: SLAB: cannot determine how compound pages are linked
> 
>       KERNEL: vmlinux
>     DUMPFILE: /var/crash/127.0.0.1-2015-11-30-13:10:23/vmcore
>         CPUS: 1
>         DATE: Mon Nov 30 13:10:14 2015
>       UPTIME: 00:00:38
> LOAD AVERAGE: 0.27, 0.08, 0.03
>        TASKS: 157
>     NODENAME: localhost
>      RELEASE: 4.4.0-rc2+
>      VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
>      MACHINE: armv7l  (unknown Mhz)
>       MEMORY: 510 MB
>        PANIC: "sysrq: SysRq : Trigger a crash"
>          PID: 1368
>      COMMAND: "bash"
>         TASK: ddbbeb40  [THREAD_INFO: ddbc4000]
>          CPU: 0
>        STATE: TASK_RUNNING (SYSRQ)
> 
> crash> bt
> PID: 1368   TASK: ddbbeb40  CPU: 0   COMMAND: "bash"
>  #0 [<c021e23c>] (sysrq_handle_crash) from [<c021e5a5>]
>  #1 [<c021e5a5>] (__handle_sysrq) from [<c021e969>]
>  #2 [<c021e969>] (write_sysrq_trigger) from [<c01067df>]
>  #3 [<c01067df>] (proc_reg_write) from [<c00ca5db>]
>  #4 [<c00ca5db>] (__vfs_write) from [<c00cac0f>]
>  #5 [<c00cac0f>] (vfs_write) from [<c00cb24b>]
>  #6 [<c00cb24b>] (sys_write) from [<c000ddc1>]
> crash>
> 
> > Can you send me the location of the vmlinux and vmcore files offline?  I can
> > try working with them tomorrow with on an x86_64 host with a crash utility
> > built with "make target=ARM".
> 
> Will save the files somewhere and let you know..
> 
> Thanks a lot
> Dave
> 


Dave,

OK, so the problem is that in CONFIG_SLAB kernels, the array_cache
structure that is pointed to by each kmem_cache.cpu_cache can now
be a vmalloc'd memory location.  That may be a new innovation for
32-bit kernels?  I don't know, but in any case, all of these error
types:

  crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410

show array_cache pointers in the vmalloc region.  The kmem_cache.cpu_cache
pointer is actually a per_cpu pointer, but on some of the kmem slab caches,
the translation of the per_cpu pointer result in a vmalloc() address instead
of a unity-mapped address.  Perhaps there's a point at which the percpu
memory allocations switch from being unity-mapped structures to being
vmalloc()'d addresses? 

In any case, at the point in time when the kmem caches are being
initialized (when the error messages get generated), the 32-bit ARM
arch has not yet set the base address of the vmalloc region in one 
if its machine-dependent variables.  And so the addresses are 
mistakenly translated as if they are unity-mapped.  The ARM arch
sets the internal vmalloc base address variable at a later point 
in time, but it can be done earlier, before the kmem cache initialization
is started.  So with this one-line patch, the dumpfile comes up cleanly:

--- arm.c.orig	2015-12-01 16:36:11.815775008 -0500
+++ arm.c	2015-12-01 16:25:44.580800293 -0500
@@ -1539,6 +1539,7 @@
 static ulong
 arm_vmalloc_start(void)
 {
+	machdep->machspec->vmalloc_start_addr = vt->high_memory;
 	return vt->high_memory;
 }
 
$ crash vmlinux vmcore-cp

crash 7.1.4rc21
Copyright (C) 2002-2014  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-elf-linux"...

      KERNEL: vmlinux                           
    DUMPFILE: vmcore-cp
        CPUS: 1
        DATE: Mon Nov 30 00:10:14 2015
      UPTIME: 00:00:38
LOAD AVERAGE: 0.27, 0.08, 0.03
       TASKS: 157
    NODENAME: localhost
     RELEASE: 4.4.0-rc2+
     VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
     MACHINE: armv7l  (unknown Mhz)
      MEMORY: 510 MB
       PANIC: "sysrq: SysRq : Trigger a crash"
         PID: 1368
     COMMAND: "bash"
        TASK: ddbbeb40  [THREAD_INFO: ddbc4000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)

crash> 

I can only test the ELF vmcore because I don't have the 32-bit lzo library
installed for my "make target=ARM" x86 binary.  But it won't be any 
different from the one above.

I'll queue the patch for crash-7.1.4.

Thanks for the report,
  Dave


--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility



[Index of Archives]     [Fedora Development]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]     [Fedora Tools]

 

Powered by Linux