Re: crash: page excluded: kernel virtual address: ffffffff81e3da50 type: "page_offset_base"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Dave,

Information update: I think I found the root cause.

During the test in last week, I always use makedumpfile, use
/etc/kdump.conf as-is:

    core_collector makedumpfile -l --message-level 1 -d 31

And now I am suggested to dump without makedumpfile by using:

    core_collector cp

I get big vmcore file of 3.5G, but crash 7.2.0++ can deal with it
without any error, whether with or without "nokaslr" kernel parameter.

So it seems makedumpfile has problem, and if it does is, I am curious
about that there is no other people report this kind of issue, because
it seems easy to reproduce.

-- 
Sincerely,
Cao jin

On 11/23/2017 10:03 PM, Dave Anderson wrote:
>> Hi Dave,
>>   Because I didn't subscribe the list, so please CC me when reply:) My
>> colleague help to forward your reply to me, so I decide to reply
>> directly in order to keep this mail in the thread other than new a thread.
>>
>> On 11/23/2017 02:04 PM, Fei, Jie/费 杰 wrote:
>>>
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: 	Re:  crash: page excluded: kernel virtual
>>> address: ffffffff81e3da50 type: "page_offset_base"
>>> Date: 	Wed, 22 Nov 2017 09:35:40 -0500
>>> From: 	Dave Anderson <anderson@xxxxxxxxxx>
>>> Reply-To: 	Discussion list for crash utility usage, maintenance and
>>> development <crash-utility@xxxxxxxxxx>
>>> To: 	Discussion list for crash utility usage, maintenance and
>>> development <crash-utility@xxxxxxxxxx>
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> Hi,
>>>>
>>>> I am using the latest crash tool & kernel 4.14 compiled from source,
>>>> and
>>>> I got the following error message. As I searched, this is fixed in
>>>> crash
>>>> 7.2.0, but I still have it here. So, is anyone has a clue?
>>>
>>> It's always going to be a crap-shoot with very recent upstream kernels,
>>> but you haven't given enough information to determine what the issue is.
>>>
>>
>> Sorry, I am to new dump/crash area.
>>
>>> If the vmcore was created by "virsh dump", and the kernel has KASLR
>>> enabled,
>>> then it's just not supported at this time.  Otherwise, perhaps the
>>> output of
>>> "crash -d8" may yield some clues.
>>>
>>
>> I am using kdump to create the vmcore. Actually, I did solve this issue
>> by add "nokaslr" to kernel parameter when I used the built-in crash of
>> Fedora. But when turn to the compiled version of my own, that solution
>> don't work. here is the output of `crash -d8`:
>>
>> $ sudo ./crash -d8 /var/crash/127.0.0.1-2017-11-22-18\:21\:42/vmcore
>> ../linux/vmlinux
>>
>> crash 7.2.0++
>> Copyright (C) 2002-2017  Red Hat, Inc.
>> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>> Copyright (C) 1999-2006  Hewlett-Packard Co
>> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>> Copyright (C) 2005, 2011  NEC Corporation
>> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>> This program is free software, covered by the GNU General Public License,
>> and you are welcome to change it and/or distribute copies of it under
>> certain conditions.  Enter "help copying" to see the conditions.
>> This program has absolutely no warranty.  Enter "help warranty" for
>> details.
>>
>> compressed kdump: header->utsname.machine: x86_64
>> compressed kdump: memory bitmap offset: 2000
>> diskdump_data:
>>           filename: /var/crash/127.0.0.1-2017-11-22-18:21:42/vmcore
>>              flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
>>                dfd: 3
>>                ofp: 0
>>       machine_type: 62 (EM_X86_64)
>>
>>             header: 1013930
>>            signature: "KDUMP   "
>>       header_version: 6
>>              utsname:
>>                sysname: Linux
>>               nodename: IAAS1
>>                release: 4.14.0
>>                version: #1 SMP Wed Nov 15 10:32:46 CST 2017
>>                machine: x86_64
>>             domainname: (none)
>>            timestamp:
>>                 tv_sec: 5a154fac
>>                tv_usec: 0
>>               status: 2 (DUMP_DH_COMPRESSED_LZO)
>>           block_size: 4096
>>         sub_hdr_size: 1
>>        bitmap_blocks: 80
>>            max_mapnr: 1310208
>>     total_ram_blocks: 0
>>        device_blocks: 0
>>       written_blocks: 0
>>          current_cpu: 0
>>              nr_cpus: 4
>>       tasks[nr_cpus]: 0
>>                       0
>>                       0
>>                       0
>>
>>         sub_header: 0 (n/a)
>>
>>   sub_header_kdump: 1014940
>>            phys_base: 0
>>           dump_level: 31 (0x1f)
>> (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
>>                split: 0
>>            start_pfn: (unused)
>>              end_pfn: (unused)
>>    offset_vmcoreinfo: 5648 (0x1610)
>>      size_vmcoreinfo: 1883 (0x75b)
>>                       OSRELEASE=4.14.0
>>                       PAGESIZE=4096
>>                       SYMBOL(init_uts_ns)=ffffffff81e10280
>>                       SYMBOL(node_online_map)=ffffffff82030e80
>>                       SYMBOL(swapper_pg_dir)=ffffffff81e09000
>>                       SYMBOL(_stext)=ffffffff81000000
>>                       SYMBOL(vmap_area_list)=ffffffff81efc470
>>                       SYMBOL(mem_section)=ffffffff82401dc0
>>                       LENGTH(mem_section)=2048
>>                       SIZE(mem_section)=16
>>                       OFFSET(mem_section.section_mem_map)=0
>>                       SIZE(page)=64
>>                       SIZE(pglist_data)=172864
>>                       SIZE(zone)=1664
>>                       SIZE(free_area)=104
>>                       SIZE(list_head)=16
>>                       SIZE(nodemask_t)=128
>>                       OFFSET(page.flags)=0
>>                       OFFSET(page._refcount)=28
>>                       OFFSET(page.mapping)=8
>>                       OFFSET(page.lru)=32
>>                       OFFSET(page._mapcount)=24
>>                       OFFSET(page.private)=48
>>                       OFFSET(page.compound_dtor)=40
>>                       OFFSET(page.compound_order)=44
>>                       OFFSET(page.compound_head)=32
>>                       OFFSET(pglist_data.node_zones)=0
>>                       OFFSET(pglist_data.nr_zones)=172192
>>                       OFFSET(pglist_data.node_start_pfn)=172200
>>                       OFFSET(pglist_data.node_spanned_pages)=172216
>>                       OFFSET(pglist_data.node_id)=172224
>>                       OFFSET(zone.free_area)=192
>>                       OFFSET(zone.vm_stat)=1472
>>                       OFFSET(zone.spanned_pages)=112
>>                       OFFSET(free_area.free_list)=0
>>                       OFFSET(list_head.next)=0
>>                       OFFSET(list_head.prev)=8
>>                       OFFSET(vmap_area.va_start)=0
>>                       OFFSET(vmap_area.list)=48
>>                       LENGTH(zone.free_area)=11
>>                       SYMBOL(log_buf)=ffffffff81e58480
>>                       SYMBOL(log_buf_len)=ffffffff81e5847c
>>                       SYMBOL(log_first_idx)=ffffffff823340f8
>>                       SYMBOL(clear_idx)=ffffffff823340cc
>>                       SYMBOL(log_next_idx)=ffffffff823340e8
>>                       SIZE(printk_log)=16
>>                       OFFSET(printk_log.ts_nsec)=0
>>                       OFFSET(printk_log.len)=8
>>                       OFFSET(printk_log.text_len)=10
>>                       OFFSET(printk_log.dict_len)=12
>>                       LENGTH(free_area.free_list)=6
>>                       NUMBER(NR_FREE_PAGES)=0
>>                       NUMBER(PG_lru)=5
>>                       NUMBER(PG_private)=12
>>                       NUMBER(PG_swapcache)=9
>>                       NUMBER(PG_slab)=8
>>                       NUMBER(PG_hwpoison)=22
>>                       NUMBER(PG_head_mask)=32768
>>                       NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
>>                       NUMBER(HUGETLB_PAGE_DTOR)=2
>>                       NUMBER(phys_base)=0
>>                       SYMBOL(init_top_pgt)=ffffffff81e09000
>>                       SYMBOL(node_data)=ffffffff8202c6c0
>>                       LENGTH(node_data)=1024
>>                       KERNELOFFSET=0
>>                       NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>>                       CRASHTIME=1511346092
>>          offset_note: 4200 (0x1068)
>>            size_note: 3332 (0xd04)
>>            notes_buf: 1015950
>>   num_prstatus_notes: 4
>>             notes[0]: 1015950 (NT_PRSTATUS)
>>                       si.signo: 0  si.code: 0  si.errno: 0
>>                       cursig: 0  sigpend: 0  sighold: 0
>>                       pid: 114  ppid: 0  pgrp: 0  sid:0
>>                       utime: 0.000000  stime: 0.000000
>>                       cutime: 0.000000  cstime: 0.000000
>>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>>                            R15: ffffffff82445b80  R14: ffffffff822f3422
>>                            R13: 0000000000000020  R12: 00000000000026f5
>>                            RBP: ffffc90000adbce0  RBX: ffffffff82445b80
>>                            R11: ffffffff822f342d  R10: 0000000000000000
>>                             R9: 000000000000000f   R8: 0000000000000000
>>                            RAX: 000000799065724b  RCX: 0000007990656c1b
>>                            RDX: 0000000000000000  RSI: 0000000000000000
>>                            RDI: 0000000000000cdd  RIP: ffffffff8188600f
>>                         RFLAGS: 0000000000000097  RSP: ffffc90000adbce0
>>                        FS_BASE: 0000000000000000
>>                        GS_BASE: 0000000000000000
>>                             CS: 0010  SS: 0018  DS: 0000
>>                             ES: 0000  FS: 0000  GS: 0000
>>             notes[1]: 1015ab4 (NT_PRSTATUS)
>>                       si.signo: 0  si.code: 0  si.errno: 0
>>                       cursig: 0  sigpend: 0  sighold: 0
>>                       pid: 2030  ppid: 0  pgrp: 0  sid:0
>>                       utime: 0.000000  stime: 0.000000
>>                       cutime: 0.000000  cstime: 0.000000
>>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>>                            R15: ffff880131999500  R14: ffffffff81f76fa0
>>                            R13: 0000000000000000  R12: 0000000000000007
>>                            RBP: ffffc90001dabdd8  RBX: 0000000000000063
>>                            R11: ffffffff822f342d  R10: 0000000000000001
>>                             R9: 0000000000000007   R8: 00000000000002d9
>>                            RAX: 000000000000000f  RCX: 0000000000000000
>>                            RDX: 0000000000000000  RSI: ffff88013fa8e138
>>                            RDI: 0000000000000063  RIP: ffffffff81532ad6
>>                         RFLAGS: 0000000000010282  RSP: ffffc90001dabdd8
>>                        FS_BASE: 00007fe58251fb80
>>                        GS_BASE: 0000000000000000
>>                             CS: 0010  SS: 0018  DS: 0000
>>                             ES: 0000  FS: 0000  GS: 0000
>>             notes[2]: 1015c18 (NT_PRSTATUS)
>>                       si.signo: 0  si.code: 0  si.errno: 0
>>                       cursig: 0  sigpend: 0  sighold: 0
>>                       pid: 0  ppid: 0  pgrp: 0  sid:0
>>                       utime: 0.000000  stime: 0.000000
>>                       cutime: 0.000000  cstime: 0.000000
>>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>>                            R15: ffffffff81f65618  R14: 0000000000000020
>>                            R13: 0000000000000004  R12: 0000000000000003
>>                            RBP: ffffc900006a3e60  RBX: 0000000000000008
>>                            R11: 000000000000298b  R10: ffffc900006a3e40
>>                             R9: 0000000000000018   R8: 00000000000054a2
>>                            RAX: 0000000000000020  RCX: 0000000000000001
>>                            RDX: 0000000000000000  RSI: ffffffff81f65480
>>                            RDI: 0000000000000002  RIP: ffffffff81895ddc
>>                         RFLAGS: 0000000000000046  RSP: ffffc900006a3e40
>>                        FS_BASE: 0000000000000000
>>                        GS_BASE: 0000000000000000
>>                             CS: 0010  SS: 0018  DS: 0000
>>                             ES: 0000  FS: 0000  GS: 0000
>>             notes[3]: 1015d7c (NT_PRSTATUS)
>>                       si.signo: 0  si.code: 0  si.errno: 0
>>                       cursig: 0  sigpend: 0  sighold: 0
>>                       pid: 0  ppid: 0  pgrp: 0  sid:0
>>                       utime: 0.000000  stime: 0.000000
>>                       cutime: 0.000000  cstime: 0.000000
>>                       ORIG_RAX: ffffffffffffffff  fpvalid: 0
>>                            R15: ffffffff81f65618  R14: 0000000000000020
>>                            R13: 0000000000000004  R12: 0000000000000003
>>                            RBP: ffffc900006abe60  RBX: 0000000000000008
>>                            R11: 000000000000037e  R10: ffffc900006abe40
>>                             R9: 0000000000000008   R8: 00000000ffffffff
>>                            RAX: 0000000000000020  RCX: 0000000000000001
>>                            RDX: 0000000000000000  RSI: ffffffff81f65480
>>                            RDI: 0000000000000003  RIP: ffffffff81895ddc
>>                         RFLAGS: 0000000000000046  RSP: ffffc900006abe40
>>                        FS_BASE: 0000000000000000
>>                        GS_BASE: 0000000000000000
>>                             CS: 0010  SS: 0018  DS: 0000
>>                             ES: 0000  FS: 0000  GS: 0000
>>        snapshot_task: 0
>>       num_qemu_notes: 0
>>         NOTE offsets: 1068 (NT_PRSTATUS)
>>                       11cc (NT_PRSTATUS)
>>                       1330 (NT_PRSTATUS)
>>                       1494 (NT_PRSTATUS)
>>     offset_eraseinfo: 0 (0x0)
>>       size_eraseinfo: 0 (0x0)
>>         start_pfn_64: (unused)
>>           end_pfn_64: (unused)
>>         max_mapnr_64: 1310208 (0x13fe00)
>>
>>        data_offset: 52000
>>         block_size: 4096
>>        block_shift: 12
>>             bitmap: 7f8ff466c010
>>         bitmap_len: 327680
>>          max_mapnr: 1310208 (0x13fe00)
>>    dumpable_bitmap: 7f8ff461b010
>>               byte: 0
>>                bit: 0
>>    compressed_page: 10470a0
>>          curbufptr: 0
>>
>>  page_cache_hdr[0]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1037090
>>         pg_hit_count: 0
>>  page_cache_hdr[1]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1038090
>>         pg_hit_count: 0
>>  page_cache_hdr[2]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1039090
>>         pg_hit_count: 0
>>  page_cache_hdr[3]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 103a090
>>         pg_hit_count: 0
>>  page_cache_hdr[4]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 103b090
>>         pg_hit_count: 0
>>  page_cache_hdr[5]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 103c090
>>         pg_hit_count: 0
>>  page_cache_hdr[6]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 103d090
>>         pg_hit_count: 0
>>  page_cache_hdr[7]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 103e090
>>         pg_hit_count: 0
>>  page_cache_hdr[8]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 103f090
>>         pg_hit_count: 0
>>  page_cache_hdr[9]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1040090
>>         pg_hit_count: 0
>> page_cache_hdr[10]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1041090
>>         pg_hit_count: 0
>> page_cache_hdr[11]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1042090
>>         pg_hit_count: 0
>> page_cache_hdr[12]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1043090
>>         pg_hit_count: 0
>> page_cache_hdr[13]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1044090
>>         pg_hit_count: 0
>> page_cache_hdr[14]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1045090
>>         pg_hit_count: 0
>> page_cache_hdr[15]:
>>             pg_flags: 0 ()
>>              pg_addr: 0
>>            pg_bufptr: 1046090
>>         pg_hit_count: 0
>>
>>     page_cache_buf: 1037090
>>        evict_index: 0
>>          evictions: 0
>>           accesses: 0
>>       cached_reads: 0
>>        valid_pages: 1036680
>> readmem: read_diskdump()
>> crash: pv_init_ops exists: ARCH_PVOPS
>> VMCOREINFO: NUMBER(phys_base): 0 -> 0
>> gdb ../linux/vmlinux
>> GNU gdb (GDB) 7.6
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-unknown-linux-gnu"...
>> GETBUF(288 -> 0)
>>   GETBUF(1500 -> 1)
>>
>>   FREEBUF(1)
>> FREEBUF(0)
>> <readmem: ffffffff81e3da50, KVADDR, "page_offset_base", 8, (FOE), d83568>
>> <read_diskdump: addr: ffffffff81e3da50 paddr: 1e3da50 cnt: 8>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 1e3da50/1e3d
>> crash: page excluded: kernel virtual address: ffffffff81e3da50  type:
>> "page_offset_base"
>>
>> --
>> Sincerely,
>> Cao jin
> 
> OK, so you seem to be describing two issues:
> 
>  (1) With KASLR enabled, the vmcore cannot be read by either
>      Fedora's 7.2.0-1 or upstream 7.2.0++.
>  (2) With KASLR disabled, the vmcore can be read by Fedora's
>      7.2.0-1, but cannot be read by 7.2.0++,
> 
> When you run crash on the live system, what happens when:
> 
>  (1) KASLR is enabled, running Fedora's 7.2.0-1
>  (2) KASLR is enabled, running upstream 7.2.0++
>  (2) KASLR is disabled, running Fedora's 7.2.0-1
>  (3) KASLR is disabled, running upstream 7.2.0++
> 
> I don't see anything unusual in the output above, presuming the
> the phys_base of 0 is correct.  When you run crash on the live
> system, what is the output of "help -m | grep phys_base"?
> 
> I need to compare the crash -d8 output above with the same
> crash -d8 output from the working session using the same
> KASLR disabled vmcore, showing at least up until the point that
> it does the readmem of "page_offset_base" above?
> 
> Also, if you can make the 2 vmcores available, please send me
> the details of how to download them in a private email to
> anderson@xxxxxxxxxx or anderson@xxxxxxxxxxxx.  If you have
> modified/rebuilt the Fedora kernel, I would also need a copy
> of the vmlinux file as well.
> 
> We are on Thanksgiving holiday until next Monday, so I won't
> be able to spend any time looking at this issue until then.
> 
> Dave
> 
> 
> 
> 
> .
> 



--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility




[Index of Archives]     [Fedora Development]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]     [Fedora Tools]

 

Powered by Linux