Re: "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Worth, Kevin wrote:
> Thanks for the explanation, Dave.
>
> I tried adding some printf's to memory.c (before the goto try_dev_kmem, then
> before and after the call to read_dev_kmem). It does in fact look like it's
> taking the read_dev_kmem path... a couple examples (a ton of these are printed,
> so if there is any specific address to look for let me know, but I presume this
> is just confirming what you suspected.

...

> Yep. Also confirmed using the printf's in previous email.
>
> crash> help -p
>      program_name: crash
>      program_path: ./crash
>   program_version: 4.0-7.2
>       gdb_version: 6.1
> ...
>               nfd: -1
>               mfd: 4
>               kfd: 11

OK, we're getting nowhere fast because of the limitations of
the /dev/mem driver.

What I've been trying to accomplish is simply this:

  on the live system:
    - find a module address whose "vtop" shows that its physical
      address is greater than 4GB.
    - rd -p <the-physical-address-greater-than-4GB> 100
    - verify that it contains "correct" data, i.e. as seen when
      you do a "p <module-virtual-address>"  (it will...)

  <crash the system>

  on the dumpfile:
    - rd -p <the-physical-address-greater-than-4GB> 100 (same as above)
    - see whether it does -- or does *not* -- contain the "correct" data
      that was seen on the live system

If the same physical address does *not* contain the same data
in the dumpfile as was there in the live system, then we can
point at the kexec/kdump operation.  We have seemingly done
so, but without being able to do it *exactly* as the steps
above because:

-  on the live system, we're relying on /dev/kmem to do the
   virtual-to-physical address translation of vmalloc addresses,
   so the resultant physical address is "hidden" from us.

If your system used the Red Hat /dev/crash driver, then the
steps above would be trivial, because the crash utility does
does the virtual-to-physical address translation of the
vmalloc addresses itself, and then reads memory using the
resultant physical address.

I will attach the crash.c and crash.h files from /dev/crash patch
we use for RHEL5's 2.6.18-based kernel.  What you would need to do
is something like this (having never done it before):

(1) Copy the attached crash.c file to your kernel's "drivers/char" directory.
(2) Copy the attached crash.h file to your kernel's "include/asm-x86_64" directory.
(3) Add this to the "drivers/char/Makefile":

+obj-m            += crash.o

(4) since 2.6.20 doesn't have a page_is_ram() function for x86_64,
    you'll have to add this to your arch/x86_64/mm/init.c file,
    and export it so the /dev/crash driver can pick it up:

int page_is_ram (unsigned long pagenr)
{
        int i;

        for (i = 0; i < e820.nr_map; i++) {
                unsigned long addr, end;

                if (e820.map[i].type != E820_RAM)       /* not usable memory */
                        continue;
                /*
                 * !!!FIXME!!! Some BIOSen report areas as RAM that
                 * are not. Notably the 640->1Mb area. We need a sanity
                 * check here.
                 */
                addr = (e820.map[i].addr+PAGE_SIZE-1) >> PAGE_SHIFT;
                end = (e820.map[i].addr+e820.map[i].size) >> PAGE_SHIFT;
                if  ((pagenr >= addr) && (pagenr < end))
                        return 1;
        }
        return 0;
}

EXPORT_SYMBOL_GPL(page_is_ram);

(5) Build the kernel and see what happens...

Other than trying that, I don't have any other suggestions.

Dave

/*
 *  linux/drivers/char/crash.c
 *
 *  Copyright (C) 2004  Dave Anderson <anderson@xxxxxxxxxx>
 *  Copyright (C) 2004  Red Hat, Inc.
 */

/******************************************************************************
 *
 *   This program is free software; you can redistribute it and/or modify
 *   it under the terms of the GNU General Public License as published by
 *   the Free Software Foundation; either version 2, or (at your option)
 *   any later version.
 *
 *   This program is distributed in the hope that it will be useful,
 *   but WITHOUT ANY WARRANTY; without even the implied warranty of
 *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *   GNU General Public License for more details.
 *
 *   You should have received a copy of the GNU General Public License
 *   along with this program; if not, write to the Free Software
 *   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
 *
 *****************************************************************************/

#include <linux/module.h>
#include <linux/types.h>
#include <linux/miscdevice.h>
#include <linux/init.h>
#include <asm/io.h>
#include <asm/uaccess.h>
#include <asm/types.h>
#include <asm/crash.h>

#define CRASH_VERSION   "1.0"

/*
 *  These are the file operation functions that allow crash utility
 *  access to physical memory.
 */

static loff_t 
crash_llseek(struct file * file, loff_t offset, int orig)
{
	switch (orig) {
	case 0:
		file->f_pos = offset;
		return file->f_pos;
	case 1:
		file->f_pos += offset;
		return file->f_pos;
	default:
		return -EINVAL;
	}
}

/*
 *  Determine the page address for an address offset value, 
 *  get a virtual address for it, and copy it out.
 *  Accesses must fit within a page.
 */
static ssize_t
crash_read(struct file *file, char *buf, size_t count, loff_t *poff)
{
	void *vaddr;
	struct page *page;
	u64 offset;
	ssize_t read;

	offset = *poff;
	if (offset >> PAGE_SHIFT != (offset+count-1) >> PAGE_SHIFT) 
		return -EINVAL;

	vaddr = map_virtual(offset, &page);
	if (!vaddr)
		return -EFAULT;

	if (copy_to_user(buf, vaddr, count)) {
		unmap_virtual(page);
		return -EFAULT;
	}
	unmap_virtual(page);

	read = count;
	*poff += read;
	return read;
}

static struct file_operations crash_fops = {
	.owner = THIS_MODULE,
	.llseek = crash_llseek,
	.read = crash_read,
};

static struct miscdevice crash_dev = {
	MISC_DYNAMIC_MINOR,
	"crash",
	&crash_fops
};

static int __init
crash_init(void)
{
	int ret;

	ret = misc_register(&crash_dev);
	if (ret) {
		printk(KERN_ERR 
		    "crash memory driver: cannot misc_register (MISC_DYNAMIC_MINOR)\n");
		goto out;
	}
	
	ret = 0;
	printk(KERN_INFO "crash memory driver: version %s\n", CRASH_VERSION);
out:
	return ret;
}

static void __exit
crash_cleanup_module(void)
{
	misc_deregister(&crash_dev);
}

module_init(crash_init);
module_exit(crash_cleanup_module);

MODULE_LICENSE("GPL");
#ifndef _ASM_X86_64_CRASH_H
#define _ASM_X86_64_CRASH_H

/*
 * linux/include/asm-x86_64/crash.h
 *
 * Copyright (c) 2004 Red Hat, Inc. All rights reserved.
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2, or (at your option)
 * any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
 *
 */

#ifdef __KERNEL__

#include <linux/mm.h>
#include <linux/highmem.h>
#include <asm/mmzone.h>

extern int page_is_ram(unsigned long);

static inline void *
map_virtual(u64 offset, struct page **pp)
{
	struct page *page;
	unsigned long pfn;
	void *vaddr;

	pfn = (unsigned long)(offset >> PAGE_SHIFT);

	if (!page_is_ram(pfn)) {
		printk(KERN_INFO
		    "crash memory driver: !page_is_ram(pfn: %lx)\n", pfn);
		return NULL;
	}

	if (!pfn_valid(pfn)) {
		printk(KERN_INFO
		    "crash memory driver: invalid pfn: %lx )\n", pfn);
		return NULL;
	}

	page = pfn_to_page(pfn);

	vaddr = kmap(page);
	if (!vaddr) {
		printk(KERN_INFO
		    "crash memory driver: pfn: %lx kmap(page: %lx) failed\n", 
			pfn, (unsigned long)page);
		return NULL;
	}

	*pp = page;
	return (vaddr + (offset & (PAGE_SIZE-1)));
}

static inline void unmap_virtual(struct page *page) 
{ 
	kunmap(page);
}

#endif /* __KERNEL__ */

#endif /* _ASM_X86_64_CRASH_H */
--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility

[Index of Archives]     [Fedora Development]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]     [Fedora Tools]

 

Powered by Linux