Re: "crash: cannot allocate any more memory!" for 2.6.27.4

"Dheeraj Sangamkar" <dheerajrs@xxxxxxxxx> · Thu, 4 Dec 2008 19:01:53 +0530

On the same machine, I am unable to live-debug the running kernel.
Here is the error I get. Please let me know if there is something I can do to get crash working.

/dheeraj/crash/crash-4.0-7.4 # crash -d 255 /boot/System.map-2.6.27.4-2-default /root/dheeraj/linux-2.6.27.4-2.1/vmlinux

crash 4.0-7.4
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited

Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.

This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.

This program has absolutely no warranty.  Enter "help warranty" for details.

crash: diskdump / compressed kdump: dump does not have panic dump header
get_live_memory_source: /dev/mem
crash: pv_init_ops exists: ARCH_PVOPS

_text: ffffffff80200000  Kernel code: 200000 -> phys_base: 0

gdb /root/dheeraj/linux-2.6.27.4-2.1/vmlinux
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.

This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(248 -> 0)
  GETBUF(1500 -> 1)

  FREEBUF(1)
FREEBUF(0)
<readmem: ffffffff804be660, KVADDR, "kernel_config_data", 32768, (ROE), 1ae5040>

/dev/mem: Operation not permitted
crash: read(/dev/mem, 4be660, 2464): 4294967295 (ffffffff)
crash: read error: kernel virtual address: ffffffff804be660  type: "kernel_config_data"
WARNING: cannot read kernel_config_data

GETBUF(248 -> 0)
FREEBUF(0)
GETBUF(16 -> 0)
<readmem: ffffffff80a14800, KVADDR, "cpu_possible_map", 16, (ROE), a49160>
/dev/mem: Operation not permitted
crash: read(/dev/mem, a14800, 16): 4294967295 (ffffffff)

crash: read error: kernel virtual address: ffffffff80a14800  type: "cpu_possible_map"
WARNING: cannot read cpu_possible_map
<readmem: ffffffff80872710, KVADDR, "cpu_present_map", 16, (ROE), a49160>

/dev/mem: Operation not permitted
crash: read(/dev/mem, 872710, 16): 4294967295 (ffffffff)
crash: read error: kernel virtual address: ffffffff80872710  type: "cpu_present_map"
WARNING: cannot read cpu_present_map

<readmem: ffffffff80846c90, KVADDR, "cpu_online_map", 16, (ROE), a49160>
/dev/mem: Operation not permitted
crash: read(/dev/mem, 846c90, 16): 4294967295 (ffffffff)
crash: read error: kernel virtual address: ffffffff80846c90  type: "cpu_online_map"

WARNING: cannot read cpu_online_map
FREEBUF(0)
<readmem: ffffffff80a74400, KVADDR, "xtime", 16, (FOE), a0d2f0>
/dev/mem: Operation not permitted
crash: read(/dev/mem, a74400, 16): 4294967295 (ffffffff)

crash: read error: kernel virtual address: ffffffff80a74400  type: "xtime"

Dheeraj

On Tue, Dec 2, 2008 at 12:00 PM, Dheeraj Sangamkar <dheerajrs@xxxxxxxxx> wrote:

Dave,Thanks a million for suggesting the patch.

Bernhard Walle,
Thanks for the patch.

All is well in my world now.

Regards,

Dheeraj

On Mon, Dec 1, 2008 at 8:43 PM, Dave Anderson <anderson@xxxxxxxxxx> wrote:

----- "Dheeraj Sangamkar" <dheerajrs@xxxxxxxxx> wrote:

> Hi,

> I am able to use crash to debug a crash dump generated when I initiate

> the crash using the sysrq-trigger in the /proc directory.

> But when I try to use crash to debug a kernel vmcore generated due to

> the insertion of my module, I get the error below.

>

> Please let me know if I am doing anything wrong or if you think that

> the dump is corrupted. I use kdump for dump collection.

> I am using a 2.6.27.4-2-default kernel.

> ---------------------------------------------------------------------------------------------------------------------------------

> /var/crash/2008-12-01-18:40 # crash System.map-default vmlinux vmcore

If the vmlinux file is the same kernel as is running on your live system,

then do not use the "System-map-default" file.  If you're not sure, then

just run "crash vmlinux vmcore", and if the vmlinux file does not match

you will get an error message.

That being said, it's highly unlikely that it is the problem at hand -- but

please avoid using System.map files if at all possible.

>

> crash 4.0-7.4

> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.

> Copyright (C) 2004, 2005, 2006 IBM Corporation

> Copyright (C) 1999-2006 Hewlett-Packard Co

> Copyright (C) 2005, 2006 Fujitsu Limited

> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.

> Copyright (C) 2005 NEC Corporation

> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.

> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.

> This program is free software, covered by the GNU General Public

> License,

> and you are welcome to change it and/or distribute copies of it under

> certain conditions. Enter "help copying" to see the conditions.

> This program has absolutely no warranty. Enter "help warranty" for

> details.

>

> GNU gdb 6.1

> Copyright 2004 Free Software Foundation, Inc.

> GDB is free software, covered by the GNU General Public License, and

> you are

> welcome to change it and/or distribute copies of it under certain

> conditions.

> Type "show copying" to see the conditions.

> There is absolutely no warranty for GDB. Type "show warranty" for

> details.

> This GDB was configured as "x86_64-unknown-linux-gnu"...

>

> please wait... (gathering module symbol data) buf_1K_used: 299

> buf_2K_used: 1

> buf_4K_used: 155

> buf_8K_used: 1

> buf_32K_used: 2

> buf_1K_ovf: 0

> buf_2K_ovf: 0

> buf_4K_ovf: 0

> buf_8K_ovf: 0

> buf_32K_ovf: 0

> buf_1K_maxuse: 2 of 10

> buf_2K_maxuse: 1 of 10

> buf_4K_maxuse: 1 of 5

> buf_8K_maxuse: 1 of 5

> buf_32K_maxuse: 1 of 1

> buf_inuse[5]: [0][0][0][0][1]

> smallest: 16

> largest: 21474854400   <=== allocation of this buffer size failed

> embedded: 2

> max_embedded: 2

> mallocs: 0

> frees: 0

> reqs/total: 459/21475603516.0

> average size: 46787807.2

>

> crash: cannot allocate any more memory!

While gathering the symbols of the currently-loaded kernel modules,

something has caused crash to attempt a bizarre memory allocation of

21474854400 (20GB+), which dutifully failed.

Most likely something has changed in the struct module that has caused

crash to go off into the weeds.  It may very well be fixed in a currently-

queued patch that Bernhard Walle sent in:

  [PATCH] Fix module size and num_symtab for 2.6.27

 https://www.redhat.com/archives/crash-utility/2008-November/msg00000.html

Try applying that patch to 4.0-7.4 and see what happens.

Dave

--

Crash-utility mailing list

Crash-utility@xxxxxxxxxx

https://www.redhat.com/mailman/listinfo/crash-utility

-- 
Simplicity of character is the natural result of profound thought.

-- 
Simplicity of character is the natural result of profound thought.

--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility