RE: [Crash-utility] Problems debugging live system with 4.0-2.24 crash

"Graham, Simon" <Simon.Graham@xxxxxxxxxxx> · Mon, 8 May 2006 10:59:37 -0400

Thanks for the info
Dave – maybe this belongs in the appropriate Xen list, but this code
seems fundamentally broken to me for a couple of reasons:

 I’m
     pretty sure that the address passed into read_mem is a physical address
     NOT a machine-address and yet the code in is_local_lowmem seems to treat
     this as an mfn (at least, I would assume that mfn_to_local_pfn would be
     doing so)
 That
     aside, it just seems wrong to me to break the operation of /dev/mem so
     that the one thing it _cant_
     return is low memory for the guest!

Time for a Xen bug I
think!

Simon

From:
crash-utility-bounces@xxxxxxxxxx [mailto:crash-utility-bounces@xxxxxxxxxx] On Behalf Of Dave Anderson

Sent: Monday, May 08, 2006 10:36
AM

To: Discussion list for crash
utility usage, maintenance and development

Subject: Re: [Crash-utility]
Problems debugging live system with 4.0-2.24 crash

"Graham, Simon" wrote: 

I've been
experimenting with the latest crash version on a system running Xen (3.0.2-2
with kernel 2.6.16) and am experiencing problems when attempting to debug the
system live; debugging of saved guest files and guest crash dumps
works well, it's just live debugging of dom0 that is problematical.

The symptom is that I get the error: 

WARNING: cannot read linux_banner string

When I debug crash, I can see that it is finding the
correct
unstripped
kernel from which it finds the address of the linux_banner variable _but_ when it attempts to read the string
from /dev/mem it gets all zeroes back. I have confirmed that the address is
correct by using gdb against /proc/kcore - gdb reads the
string quite nicely from the virtual address (happens to be 0xc0450000,
I can see that crash translates this to physical address 450000 and gets all
zeroes when it reads this location from /dev/mem).

Hi
Graham, 

The xen /dev/mem driver read_mem() function does this:

if ((v = ioremap(p, sz)) == NULL) { 

/* 

* Some programs (e.g., dmidecode) groove off into weird RAM 

* areas where no tables can possibly exist (because Xen will

* have stomped on them!). These programs get rather upset if

* we let them know that Xen failed their access, so we fake 

* out a read of all zeroes. :-) 

*/ 

if (clear_user(buf, count)) 

return -EFAULT; 

read += count; 

break; 

} 

when ioremap() returns NULL.  Here's the
NULL-returner in __ioremap(): 

        /*

* Don't allow anybody to remap normal RAM that we're using..

*/ 

if (is_local_lowmem(phys_addr)) { 

char *t_addr, *t_end; 

struct page *page; 

t_addr = bus_to_virt(phys_addr); 

t_end = t_addr + (size - 1); 

for(page = virt_to_page(t_addr); page <= virt_to_page(t_end); page++)

if(!PageReserved(page)) 

return NULL; 

domid = DOMID_SELF; 

} 

So /dev/mem cannot be used.  So to get crash to
work on a live 

x86 xen0
system (FC5), I've had to use the RHEL4 crash.c module 

(/dev/crash)
to make an end-run around /dev/mem.  RHEL4's x86 and 

x86_64
/dev/mem driver only allows reads of the first 256 pages of 

physical
memory.  And now Xen has the restriction above. 

FC5 has a CONFIG_CRASH configurable:

$ diff kernel-2.6.15-i686-guest.config.orig
kernel-2.6.15-i686-guest.config 

2526c2526

< #
CONFIG_CRASH is not set 

---

>
CONFIG_CRASH=m 

2965d2964

< #
CONFIG_CRASH is not set 

$

If you're building from an upstream source, you could
build the 

crash.c
module from FC5/RHEL4 into that kernel, or perhaps 

patch the
xen /dev/mem driver above to remove the local_lowmem() 

restriction,
or something like that... 

Dave 

I'm pretty sure that the various files in /boot for
dom0 all match the running system. Output from `crash
-d2'
attached in case its useful... 

So - the question is
-
what's
the obvious thing I am missing! 

Thanks, 

Simon 

crash 4.0-2.24

Copyright (C) 2002, 2003,
2004, 2005, 2006  Red Hat, Inc. 

Copyright (C) 2004, 2005,
2006  IBM Corporation 

Copyright (C)
1999-2006  Hewlett-Packard Co 

Copyright (C) 2005 
Fujitsu Limited 

Copyright (C) 2005  NEC
Corporation 

Copyright (C) 1999,
2002  Silicon Graphics, Inc. 

Copyright (C) 1999, 2000,
2001, 2002  Mission Critical Linux, Inc. 

This program is free
software, covered by the GNU General Public License, 

and you are welcome to
change it and/or distribute copies of it under 

certain conditions. 
Enter "help copying" to see the conditions. 

This program has absolutely
no warranty.  Enter "help warranty" for details. 

find_booted_kernel: search
for [Linux version 2.6.16-xen0 (sgraham@xxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4
20050721 (Red Hat 3.4.4-2)) #3 Sat May 6 19:04:09 EDT 2006 

] 

mount_points[0]: / (83a1ac0)

mount_points[1]: /proc
(83a1ad0) 

mount_points[2]: /dev
(83a1ae0) 

mount_points[3]: / (83a1af0)

mount_points[4]: /dev
(83a1b00) 

mount_points[5]: /proc
(83a1b10) 

mount_points[6]: /sys
(83a1b20) 

mount_points[7]: /dev/pts
(83a1b30) 

mount_points[8]: /boot
(83a1b48) 

mount_points[9]: /dev/shm
(83a1b58) 

searchdirs[9]:
/usr/lib/debug/lib/modules/2.6.16-xen0/ 

searchdirs[0]:
/usr/src/linux/ 

searchdirs[1]: /boot/

searchdirs[2]:
/boot/efi/redhat 

searchdirs[3]:
/boot/efi/EFI/redhat 

searchdirs[4]: /

searchdirs[5]:
/usr/src/kernels/ 

searchdirs[6]:
/usr/src/debug/ 

searchdirs[7]:
/usr/src/redhat/BUILD/kernel-2.6.16/linux/ 

searchdirs[8]:
/usr/src/redhat/BUILD/kernel-2.6.16/linux-2.6.16/ 

find_booted_kernel: check:
/boot/xen-syms-3.0.2-2 

find_booted_kernel: check:
/boot/vmlinux-syms-2.6.16-xenU 

find_booted_kernel: check:
/boot/vmlinux-syms-2.6.16-xen0 

find_booted_kernel: found:
/boot/vmlinux-syms-2.6.16-xen0 

get_live_memory_source:
/dev/mem 

/proc/version:

Linux version 2.6.16-xen0
(sgraham@xxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3
Sat May 6 19:04:09 EDT 2006 

/boot/vmlinux-syms-2.6.16-xen0:

Linux version 2.6.16-xen0
(sgraham@xxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3
Sat May 6 19:04:09 EDT 2006 

gdb
/boot/vmlinux-syms-2.6.16-xen0  

GNU gdb 6.1 

GDB is free software,
covered by the GNU General Public License, and you are 

welcome to change it and/or
distribute copies of it under certain conditions. 

Type "show
copying" to see the conditions. 

There is absolutely no
warranty for GDB.  Type "show warranty" for details.

This GDB was configured as
"i686-pc-linux-gnu"... 

WARNING: cannot read
linux_banner string 

/proc/version:

Linux version 2.6.16-xen0
(sgraham@xxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3
Sat May 6 19:04:09 EDT 2006 

linux_banner: 

crash:
/boot/vmlinux-syms-2.6.16-xen0 and /dev/mem do not match! 

Usage: 

  crash [-h
[opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]

Enter "crash -h"
for details. 

--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility