Hi Dave, Information update: I think I found the root cause. During the test in last week, I always use makedumpfile, use /etc/kdump.conf as-is: core_collector makedumpfile -l --message-level 1 -d 31 And now I am suggested to dump without makedumpfile by using: core_collector cp I get big vmcore file of 3.5G, but crash 7.2.0++ can deal with it without any error, whether with or without "nokaslr" kernel parameter. So it seems makedumpfile has problem, and if it does is, I am curious about that there is no other people report this kind of issue, because it seems easy to reproduce. -- Sincerely, Cao jin On 11/23/2017 10:03 PM, Dave Anderson wrote: >> Hi Dave, >> Â Because I didn't subscribe the list, so please CC me when reply:) My >> colleague help to forward your reply to me, so I decide to reply >> directly in order to keep this mail in the thread other than new a thread. >> >> On 11/23/2017 02:04 PM, Fei, Jie/è´¹ æ° wrote: >>> >>> >>> >>> -------- Forwarded Message -------- >>> Subject: Re: crash: page excluded: kernel virtual >>> address: ffffffff81e3da50 type: "page_offset_base" >>> Date: Wed, 22 Nov 2017 09:35:40 -0500 >>> From: Dave Anderson <anderson@xxxxxxxxxx> >>> Reply-To: Discussion list for crash utility usage, maintenance and >>> development <crash-utility@xxxxxxxxxx> >>> To: Discussion list for crash utility usage, maintenance and >>> development <crash-utility@xxxxxxxxxx> >>> >>> >>> >>> ----- Original Message ----- >>>> Hi, >>>> >>>> I am using the latest crash tool & kernel 4.14 compiled from source, >>>> and >>>> I got the following error message. As I searched, this is fixed in >>>> crash >>>> 7.2.0, but I still have it here. So, is anyone has a clue? >>> >>> It's always going to be a crap-shoot with very recent upstream kernels, >>> but you haven't given enough information to determine what the issue is. >>> >> >> Sorry, I am to new dump/crash area. >> >>> If the vmcore was created by "virsh dump", and the kernel has KASLR >>> enabled, >>> then it's just not supported at this time. Otherwise, perhaps the >>> output of >>> "crash -d8" may yield some clues. >>> >> >> I am using kdump to create the vmcore. Actually, I did solve this issue >> by add "nokaslr" to kernel parameter when I used the built-in crash of >> Fedora. But when turn to the compiled version of my own, that solution >> don't work. here is the output of `crash -d8`: >> >> $ sudo ./crash -d8 /var/crash/127.0.0.1-2017-11-22-18\:21\:42/vmcore >> ../linux/vmlinux >> >> crash 7.2.0++ >> Copyright (C) 2002-2017 Red Hat, Inc. >> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation >> Copyright (C) 1999-2006 Hewlett-Packard Co >> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited >> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. >> Copyright (C) 2005, 2011 NEC Corporation >> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. >> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. >> This program is free software, covered by the GNU General Public License, >> and you are welcome to change it and/or distribute copies of it under >> certain conditions. Enter "help copying" to see the conditions. >> This program has absolutely no warranty. Enter "help warranty" for >> details. >> >> compressed kdump: header->utsname.machine: x86_64 >> compressed kdump: memory bitmap offset: 2000 >> diskdump_data: >> filename: /var/crash/127.0.0.1-2017-11-22-18:21:42/vmcore >> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) >> dfd: 3 >> ofp: 0 >> machine_type: 62 (EM_X86_64) >> >> header: 1013930 >> signature: "KDUMP " >> header_version: 6 >> utsname: >> sysname: Linux >> nodename: IAAS1 >> release: 4.14.0 >> version: #1 SMP Wed Nov 15 10:32:46 CST 2017 >> machine: x86_64 >> domainname: (none) >> timestamp: >> tv_sec: 5a154fac >> tv_usec: 0 >> status: 2 (DUMP_DH_COMPRESSED_LZO) >> block_size: 4096 >> sub_hdr_size: 1 >> bitmap_blocks: 80 >> max_mapnr: 1310208 >> total_ram_blocks: 0 >> device_blocks: 0 >> written_blocks: 0 >> current_cpu: 0 >> nr_cpus: 4 >> tasks[nr_cpus]: 0 >> 0 >> 0 >> 0 >> >> sub_header: 0 (n/a) >> >> sub_header_kdump: 1014940 >> phys_base: 0 >> dump_level: 31 (0x1f) >> (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE) >> split: 0 >> start_pfn: (unused) >> end_pfn: (unused) >> offset_vmcoreinfo: 5648 (0x1610) >> size_vmcoreinfo: 1883 (0x75b) >> OSRELEASE=4.14.0 >> PAGESIZE=4096 >> SYMBOL(init_uts_ns)=ffffffff81e10280 >> SYMBOL(node_online_map)=ffffffff82030e80 >> SYMBOL(swapper_pg_dir)=ffffffff81e09000 >> SYMBOL(_stext)=ffffffff81000000 >> SYMBOL(vmap_area_list)=ffffffff81efc470 >> SYMBOL(mem_section)=ffffffff82401dc0 >> LENGTH(mem_section)=2048 >> SIZE(mem_section)=16 >> OFFSET(mem_section.section_mem_map)=0 >> SIZE(page)=64 >> SIZE(pglist_data)=172864 >> SIZE(zone)=1664 >> SIZE(free_area)=104 >> SIZE(list_head)=16 >> SIZE(nodemask_t)=128 >> OFFSET(page.flags)=0 >> OFFSET(page._refcount)=28 >> OFFSET(page.mapping)=8 >> OFFSET(page.lru)=32 >> OFFSET(page._mapcount)=24 >> OFFSET(page.private)=48 >> OFFSET(page.compound_dtor)=40 >> OFFSET(page.compound_order)=44 >> OFFSET(page.compound_head)=32 >> OFFSET(pglist_data.node_zones)=0 >> OFFSET(pglist_data.nr_zones)=172192 >> OFFSET(pglist_data.node_start_pfn)=172200 >> OFFSET(pglist_data.node_spanned_pages)=172216 >> OFFSET(pglist_data.node_id)=172224 >> OFFSET(zone.free_area)=192 >> OFFSET(zone.vm_stat)=1472 >> OFFSET(zone.spanned_pages)=112 >> OFFSET(free_area.free_list)=0 >> OFFSET(list_head.next)=0 >> OFFSET(list_head.prev)=8 >> OFFSET(vmap_area.va_start)=0 >> OFFSET(vmap_area.list)=48 >> LENGTH(zone.free_area)=11 >> SYMBOL(log_buf)=ffffffff81e58480 >> SYMBOL(log_buf_len)=ffffffff81e5847c >> SYMBOL(log_first_idx)=ffffffff823340f8 >> SYMBOL(clear_idx)=ffffffff823340cc >> SYMBOL(log_next_idx)=ffffffff823340e8 >> SIZE(printk_log)=16 >> OFFSET(printk_log.ts_nsec)=0 >> OFFSET(printk_log.len)=8 >> OFFSET(printk_log.text_len)=10 >> OFFSET(printk_log.dict_len)=12 >> LENGTH(free_area.free_list)=6 >> NUMBER(NR_FREE_PAGES)=0 >> NUMBER(PG_lru)=5 >> NUMBER(PG_private)=12 >> NUMBER(PG_swapcache)=9 >> NUMBER(PG_slab)=8 >> NUMBER(PG_hwpoison)=22 >> NUMBER(PG_head_mask)=32768 >> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128 >> NUMBER(HUGETLB_PAGE_DTOR)=2 >> NUMBER(phys_base)=0 >> SYMBOL(init_top_pgt)=ffffffff81e09000 >> SYMBOL(node_data)=ffffffff8202c6c0 >> LENGTH(node_data)=1024 >> KERNELOFFSET=0 >> NUMBER(KERNEL_IMAGE_SIZE)=1073741824 >> CRASHTIME=1511346092 >> offset_note: 4200 (0x1068) >> size_note: 3332 (0xd04) >> notes_buf: 1015950 >> num_prstatus_notes: 4 >> notes[0]: 1015950 (NT_PRSTATUS) >> si.signo: 0 si.code: 0 si.errno: 0 >> cursig: 0 sigpend: 0 sighold: 0 >> pid: 114 ppid: 0 pgrp: 0 sid:0 >> utime: 0.000000 stime: 0.000000 >> cutime: 0.000000 cstime: 0.000000 >> ORIG_RAX: ffffffffffffffff fpvalid: 0 >> R15: ffffffff82445b80 R14: ffffffff822f3422 >> R13: 0000000000000020 R12: 00000000000026f5 >> RBP: ffffc90000adbce0 RBX: ffffffff82445b80 >> R11: ffffffff822f342d R10: 0000000000000000 >> R9: 000000000000000f R8: 0000000000000000 >> RAX: 000000799065724b RCX: 0000007990656c1b >> RDX: 0000000000000000 RSI: 0000000000000000 >> RDI: 0000000000000cdd RIP: ffffffff8188600f >> RFLAGS: 0000000000000097 RSP: ffffc90000adbce0 >> FS_BASE: 0000000000000000 >> GS_BASE: 0000000000000000 >> CS: 0010 SS: 0018 DS: 0000 >> ES: 0000 FS: 0000 GS: 0000 >> notes[1]: 1015ab4 (NT_PRSTATUS) >> si.signo: 0 si.code: 0 si.errno: 0 >> cursig: 0 sigpend: 0 sighold: 0 >> pid: 2030 ppid: 0 pgrp: 0 sid:0 >> utime: 0.000000 stime: 0.000000 >> cutime: 0.000000 cstime: 0.000000 >> ORIG_RAX: ffffffffffffffff fpvalid: 0 >> R15: ffff880131999500 R14: ffffffff81f76fa0 >> R13: 0000000000000000 R12: 0000000000000007 >> RBP: ffffc90001dabdd8 RBX: 0000000000000063 >> R11: ffffffff822f342d R10: 0000000000000001 >> R9: 0000000000000007 R8: 00000000000002d9 >> RAX: 000000000000000f RCX: 0000000000000000 >> RDX: 0000000000000000 RSI: ffff88013fa8e138 >> RDI: 0000000000000063 RIP: ffffffff81532ad6 >> RFLAGS: 0000000000010282 RSP: ffffc90001dabdd8 >> FS_BASE: 00007fe58251fb80 >> GS_BASE: 0000000000000000 >> CS: 0010 SS: 0018 DS: 0000 >> ES: 0000 FS: 0000 GS: 0000 >> notes[2]: 1015c18 (NT_PRSTATUS) >> si.signo: 0 si.code: 0 si.errno: 0 >> cursig: 0 sigpend: 0 sighold: 0 >> pid: 0 ppid: 0 pgrp: 0 sid:0 >> utime: 0.000000 stime: 0.000000 >> cutime: 0.000000 cstime: 0.000000 >> ORIG_RAX: ffffffffffffffff fpvalid: 0 >> R15: ffffffff81f65618 R14: 0000000000000020 >> R13: 0000000000000004 R12: 0000000000000003 >> RBP: ffffc900006a3e60 RBX: 0000000000000008 >> R11: 000000000000298b R10: ffffc900006a3e40 >> R9: 0000000000000018 R8: 00000000000054a2 >> RAX: 0000000000000020 RCX: 0000000000000001 >> RDX: 0000000000000000 RSI: ffffffff81f65480 >> RDI: 0000000000000002 RIP: ffffffff81895ddc >> RFLAGS: 0000000000000046 RSP: ffffc900006a3e40 >> FS_BASE: 0000000000000000 >> GS_BASE: 0000000000000000 >> CS: 0010 SS: 0018 DS: 0000 >> ES: 0000 FS: 0000 GS: 0000 >> notes[3]: 1015d7c (NT_PRSTATUS) >> si.signo: 0 si.code: 0 si.errno: 0 >> cursig: 0 sigpend: 0 sighold: 0 >> pid: 0 ppid: 0 pgrp: 0 sid:0 >> utime: 0.000000 stime: 0.000000 >> cutime: 0.000000 cstime: 0.000000 >> ORIG_RAX: ffffffffffffffff fpvalid: 0 >> R15: ffffffff81f65618 R14: 0000000000000020 >> R13: 0000000000000004 R12: 0000000000000003 >> RBP: ffffc900006abe60 RBX: 0000000000000008 >> R11: 000000000000037e R10: ffffc900006abe40 >> R9: 0000000000000008 R8: 00000000ffffffff >> RAX: 0000000000000020 RCX: 0000000000000001 >> RDX: 0000000000000000 RSI: ffffffff81f65480 >> RDI: 0000000000000003 RIP: ffffffff81895ddc >> RFLAGS: 0000000000000046 RSP: ffffc900006abe40 >> FS_BASE: 0000000000000000 >> GS_BASE: 0000000000000000 >> CS: 0010 SS: 0018 DS: 0000 >> ES: 0000 FS: 0000 GS: 0000 >> snapshot_task: 0 >> num_qemu_notes: 0 >> NOTE offsets: 1068 (NT_PRSTATUS) >> 11cc (NT_PRSTATUS) >> 1330 (NT_PRSTATUS) >> 1494 (NT_PRSTATUS) >> offset_eraseinfo: 0 (0x0) >> size_eraseinfo: 0 (0x0) >> start_pfn_64: (unused) >> end_pfn_64: (unused) >> max_mapnr_64: 1310208 (0x13fe00) >> >> data_offset: 52000 >> block_size: 4096 >> block_shift: 12 >> bitmap: 7f8ff466c010 >> bitmap_len: 327680 >> max_mapnr: 1310208 (0x13fe00) >> dumpable_bitmap: 7f8ff461b010 >> byte: 0 >> bit: 0 >> compressed_page: 10470a0 >> curbufptr: 0 >> >> page_cache_hdr[0]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1037090 >> pg_hit_count: 0 >> page_cache_hdr[1]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1038090 >> pg_hit_count: 0 >> page_cache_hdr[2]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1039090 >> pg_hit_count: 0 >> page_cache_hdr[3]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 103a090 >> pg_hit_count: 0 >> page_cache_hdr[4]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 103b090 >> pg_hit_count: 0 >> page_cache_hdr[5]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 103c090 >> pg_hit_count: 0 >> page_cache_hdr[6]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 103d090 >> pg_hit_count: 0 >> page_cache_hdr[7]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 103e090 >> pg_hit_count: 0 >> page_cache_hdr[8]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 103f090 >> pg_hit_count: 0 >> page_cache_hdr[9]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1040090 >> pg_hit_count: 0 >> page_cache_hdr[10]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1041090 >> pg_hit_count: 0 >> page_cache_hdr[11]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1042090 >> pg_hit_count: 0 >> page_cache_hdr[12]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1043090 >> pg_hit_count: 0 >> page_cache_hdr[13]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1044090 >> pg_hit_count: 0 >> page_cache_hdr[14]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1045090 >> pg_hit_count: 0 >> page_cache_hdr[15]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 1046090 >> pg_hit_count: 0 >> >> page_cache_buf: 1037090 >> evict_index: 0 >> evictions: 0 >> accesses: 0 >> cached_reads: 0 >> valid_pages: 1036680 >> readmem: read_diskdump() >> crash: pv_init_ops exists: ARCH_PVOPS >> VMCOREINFO: NUMBER(phys_base): 0 -> 0 >> gdb ../linux/vmlinux >> GNU gdb (GDB) 7.6 >> Copyright (C) 2013 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later >> <http://gnu.org/licenses/gpl.html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-unknown-linux-gnu"... >> GETBUF(288 -> 0) >> GETBUF(1500 -> 1) >> >> FREEBUF(1) >> FREEBUF(0) >> <readmem: ffffffff81e3da50, KVADDR, "page_offset_base", 8, (FOE), d83568> >> <read_diskdump: addr: ffffffff81e3da50 paddr: 1e3da50 cnt: 8> >> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 1e3da50/1e3d >> crash: page excluded: kernel virtual address: ffffffff81e3da50 type: >> "page_offset_base" >> >> -- >> Sincerely, >> Cao jin > > OK, so you seem to be describing two issues: > > (1) With KASLR enabled, the vmcore cannot be read by either > Fedora's 7.2.0-1 or upstream 7.2.0++. > (2) With KASLR disabled, the vmcore can be read by Fedora's > 7.2.0-1, but cannot be read by 7.2.0++, > > When you run crash on the live system, what happens when: > > (1) KASLR is enabled, running Fedora's 7.2.0-1 > (2) KASLR is enabled, running upstream 7.2.0++ > (2) KASLR is disabled, running Fedora's 7.2.0-1 > (3) KASLR is disabled, running upstream 7.2.0++ > > I don't see anything unusual in the output above, presuming the > the phys_base of 0 is correct. When you run crash on the live > system, what is the output of "help -m | grep phys_base"? > > I need to compare the crash -d8 output above with the same > crash -d8 output from the working session using the same > KASLR disabled vmcore, showing at least up until the point that > it does the readmem of "page_offset_base" above? > > Also, if you can make the 2 vmcores available, please send me > the details of how to download them in a private email to > anderson@xxxxxxxxxx or anderson@xxxxxxxxxxxx. If you have > modified/rebuilt the Fedora kernel, I would also need a copy > of the vmlinux file as well. > > We are on Thanksgiving holiday until next Monday, so I won't > be able to spend any time looking at this issue until then. > > Dave > > > > > . > -- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility
