Hello, On 7/20/20 8:04 AM, Agrain Patrick wrote: > Hello all, > > I've upgraded my Centos 8 VM to the latest kernel (4.18.0-193.6.3.el8_2.x86_64). > I have then installed the kernel-debug packages (and dependencies), as suggested below. > > Now, I have following kernels in /boot: > - vmlinuz-4.18.0-193.6.3.el8_2.x86_64 > - vmlinuz-4.18.0-193.6.3.el8_2.x86_64+debug This is a kernel-debug pacakge, not kernel-debuginfo package. I think you can get the kernel-debuginfo here: http://debuginfo.centos.org/8/x86_64/Packages/ Probably you need to install kernel-debuginfo-common-x86_64 package as well. After the debuginfo package is installed, you'll see the vmlinux as follows: /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux Then: crash /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux vmcore Could you try it? Thanks, Masa > And a vmlinuz file in /lib/modules/4.18.0-193.6.3.el8_2.x86_64+debug > > Then, I reboot the VM on this latest kernel and extracted the vmlinux files with the extract-vmlinux script from a CentOS 4.18.0 kernel source tree. > A last, the 'echo c > /proc/sysrq-trigger' creates the /var/crash/vmcore file. > > Exporting the vmcore, vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug and vmlinux (from /lib/modules) on another CentOS machine and executing: > /usr/bin/crash -d 15 vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug vmcore > returns: > crash: vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug: no .gnu_debuglink section > > Note: same result when using the exported vmlinux from /lib/modules directory. > > Still not driving me nut... but not far away... > > To be continued... > Thanks. > Best regards, > Patrick > > > -----Message d'origine----- > De : crash-utility-bounces@xxxxxxxxxx [mailto:crash-utility-bounces@xxxxxxxxxx] De la part de Kazuhito Hagio > Envoyé : jeudi 9 juillet 2020 07:00 > À : Discussion list for crash utility usage, maintenance and development <crash-utility@xxxxxxxxxx> > Objet : EXT: Re: EXT: Re: crash start in CentOS 8 > > > ** External email - Please consider with caution ** > > > Hi Patrick, > >> OSRELEASE=4.18.0-147.el8.x86_64 > > Hmm, I think you might boot the virtual machine with the original (unmodified) CentOS 8 kernel, because the address of init_uts_ns looks to be its one: > >> SYMBOL(init_uts_ns)=ffffffffb5a12540 >> KERNELOFFSET=33800000 > > I happen to have a CentOS 8 machine.. > > # grep 'D init_uts_ns' /boot/System.map-4.18.0-147.el8.x86_64 > ffffffff82212540 D init_uts_ns > # printf "%16llx\n" $(( 0xffffffff82212540 + 0x33800000 )) > ffffffffb5a12540 > > while your vmcore.debuginfo has the different address: > >> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE), d71b7c> > > So, which kernel do you actually want to use on the virtual machine? > (1) The original CentOS 8 kernel > (2) The kernel you built > > If it's (1), you can just install the kernel-debuginfo package corresponding to the original CentOS 8 kernel into the machine on which you want to analyze vmcores and execute: > # crash /usr/lib/debug/lib/modules/4.18.0-147.el8.x86_64/vmlinux vmcore > > If it's (2), you need to boot the VM with the kernel and make a crash. > > Could you check this first? > > Thanks, > Kazu > > > On Mon, Jul 6, 2020 at 10:03 PM Agrain Patrick <patrick.agrain@xxxxxxxxxxxxxxxxx> wrote: >> >> Hello all, >> >> Back on this topic: >> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8. >> I made a crash on this same VM. >> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal machine. >> >> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces following logs, but still failing to get the crash> prompt. >> Still KASLR ? >> >> Thanks in advance for any information. >> Best regards, >> Patrick Agrain >> >> -- Logs: >> crash 7.2.6-2.el8 >> Copyright (C) 2002-2019 Red Hat, Inc. >> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) >> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 >> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. >> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, >> 2007 Silicon Graphics, Inc. >> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. >> This program is free software, covered by the GNU General Public >> License, and you are welcome to change it and/or distribute copies of >> it under certain conditions. Enter "help copying" to see the conditions. >> This program has absolutely no warranty. Enter "help warranty" for details. >> >> compressed kdump: header->utsname.machine: x86_64 compressed kdump: >> memory bitmap offset: 2000 >> diskdump_data: >> filename: vmcore >> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) >> dfd: 3 >> ofp: 0 >> machine_type: 62 (EM_X86_64) >> >> header: 212b5d0 >> signature: "KDUMP " >> header_version: 6 >> utsname: >> sysname: Linux >> nodename: localhost.localdomain >> release: 4.18.0-147.el8.x86_64 >> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019 >> machine: x86_64 >> domainname: (none) >> timestamp: >> tv_sec: 5ee72d23 >> tv_usec: 0 >> status: 2 (DUMP_DH_COMPRESSED_LZO) >> block_size: 4096 >> sub_hdr_size: 1 >> bitmap_blocks: 32 >> max_mapnr: 524288 >> total_ram_blocks: 0 >> device_blocks: 0 >> written_blocks: 0 >> current_cpu: 0 >> nr_cpus: 1 >> tasks[nr_cpus]: 0 >> >> sub_header: 0 (n/a) >> >> sub_header_kdump: 212c5e0 >> phys_base: 7c00000 >> dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE) >> split: 0 >> start_pfn: (unused) >> end_pfn: (unused) >> offset_vmcoreinfo: 4580 (0x11e4) >> size_vmcoreinfo: 2025 (0x7e9) >> OSRELEASE=4.18.0-147.el8.x86_64 >> PAGESIZE=4096 >> SYMBOL(init_uts_ns)=ffffffffb5a12540 >> SYMBOL(node_online_map)=ffffffffb5c06d60 >> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000 >> SYMBOL(_stext)=ffffffffb4800000 >> SYMBOL(vmap_area_list)=ffffffffb5ad2a90 >> SYMBOL(mem_section)=ffff89df7ffd2000 >> LENGTH(mem_section)=2048 >> SIZE(mem_section)=16 >> OFFSET(mem_section.section_mem_map)=0 >> SIZE(page)=64 >> SIZE(pglist_data)=171968 >> SIZE(zone)=1472 >> SIZE(free_area)=88 >> SIZE(list_head)=16 >> SIZE(nodemask_t)=128 >> OFFSET(page.flags)=0 >> OFFSET(page._refcount)=52 >> OFFSET(page.mapping)=24 >> OFFSET(page.lru)=8 >> OFFSET(page._mapcount)=48 >> OFFSET(page.private)=40 >> OFFSET(page.compound_dtor)=16 >> OFFSET(page.compound_order)=17 >> OFFSET(page.compound_head)=8 >> OFFSET(pglist_data.node_zones)=0 >> OFFSET(pglist_data.nr_zones)=171232 >> OFFSET(pglist_data.node_start_pfn)=171240 >> OFFSET(pglist_data.node_spanned_pages)=171256 >> OFFSET(pglist_data.node_id)=171264 >> OFFSET(zone.free_area)=192 >> OFFSET(zone.vm_stat)=1296 >> OFFSET(zone.spanned_pages)=112 >> OFFSET(free_area.free_list)=0 >> OFFSET(list_head.next)=0 >> OFFSET(list_head.prev)=8 >> OFFSET(vmap_area.va_start)=0 >> OFFSET(vmap_area.list)=48 >> LENGTH(zone.free_area)=11 >> SYMBOL(log_buf)=ffffffffb5a5a280 >> SYMBOL(log_buf_len)=ffffffffb5a5a27c >> SYMBOL(log_first_idx)=ffffffffb633a770 >> SYMBOL(clear_idx)=ffffffffb633a744 >> SYMBOL(log_next_idx)=ffffffffb633a760 >> SIZE(printk_log)=16 >> OFFSET(printk_log.ts_nsec)=0 >> OFFSET(printk_log.len)=8 >> OFFSET(printk_log.text_len)=10 >> OFFSET(printk_log.dict_len)=12 >> LENGTH(free_area.free_list)=5 >> NUMBER(NR_FREE_PAGES)=0 >> NUMBER(PG_lru)=5 >> NUMBER(PG_private)=12 >> NUMBER(PG_swapcache)=9 >> NUMBER(PG_swapbacked)=18 >> NUMBER(PG_slab)=8 >> NUMBER(PG_hwpoison)=22 >> NUMBER(PG_head_mask)=32768 >> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129 >> NUMBER(HUGETLB_PAGE_DTOR)=2 >> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257 >> NUMBER(phys_base)=130023424 >> SYMBOL(init_top_pgt)=ffffffffb5a0a000 >> NUMBER(pgtable_l5_enabled)=0 >> SYMBOL(node_data)=ffffffffb5c02580 >> LENGTH(node_data)=1024 >> KERNELOFFSET=33800000 >> NUMBER(KERNEL_IMAGE_SIZE)=1073741824 >> NUMBER(sme_mask)=0 >> CRASHTIME=1592208675 >> offset_note: 4200 (0x1068) >> size_note: 2408 (0x968) >> notes_buf: 2129460 >> num_vmcoredd_notes: 0 >> num_prstatus_notes: 1 >> notes[0]: 2129460 (NT_PRSTATUS) >> si.signo: 0 si.code: 0 si.errno: 0 >> cursig: 0 sigpend: 0 sighold: 0 >> pid: 2846 ppid: 0 pgrp: 0 sid:0 >> utime: 0.000000 stime: 0.000000 >> cutime: 0.000000 cstime: 0.000000 >> ORIG_RAX: ffffffffffffffff fpvalid: 0 >> R15: 0000000000000000 R14: 00005616bb1208f0 >> R13: ffffffffb5b36fc0 R12: 0000000000000000 >> RBP: 0000000000000004 RBX: 0000000000000063 >> R11: 6873617263206120 R10: 7265676769725420 >> R9: 54203a2071527379 R8: 000000000000056a >> RAX: ffffffffb4d12d10 RCX: 0000000000000006 >> RDX: 0000000000000000 RSI: 0000000000000086 >> RDI: 0000000000000063 RIP: ffffffffb4d12d22 >> RFLAGS: 0000000000010246 RSP: ffffb03903147e80 >> FS_BASE: 00007f5d6e06f740 >> GS_BASE: 0000000000000000 >> CS: 0010 SS: 0018 DS: 0000 >> ES: 0000 FS: 0000 GS: 0000 >> snapshot_task: 0 >> num_qemu_notes: 0 >> NOTE offsets: 1068 (NT_PRSTATUS) >> offset_eraseinfo: 0 (0x0) >> size_eraseinfo: 0 (0x0) >> start_pfn_64: (unused) >> end_pfn_64: (unused) >> max_mapnr_64: 524288 (0x80000) >> >> data_offset: 22000 >> block_size: 4096 >> block_shift: 12 >> bitmap: 212f190 >> bitmap_len: 131072 >> max_mapnr: 524288 (0x80000) >> dumpable_bitmap: 7f937ad93010 >> byte: 0 >> bit: 0 >> compressed_page: 212d5f0 >> curbufptr: 0 >> >> page_cache_hdr[0]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 217f1d0 >> pg_hit_count: 0 >> page_cache_hdr[1]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21801d0 >> pg_hit_count: 0 >> page_cache_hdr[2]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21811d0 >> pg_hit_count: 0 >> page_cache_hdr[3]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21821d0 >> pg_hit_count: 0 >> page_cache_hdr[4]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21831d0 >> pg_hit_count: 0 >> page_cache_hdr[5]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21841d0 >> pg_hit_count: 0 >> page_cache_hdr[6]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21851d0 >> pg_hit_count: 0 >> page_cache_hdr[7]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21861d0 >> pg_hit_count: 0 >> page_cache_hdr[8]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21871d0 >> pg_hit_count: 0 >> page_cache_hdr[9]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21881d0 >> pg_hit_count: 0 >> page_cache_hdr[10]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 21891d0 >> pg_hit_count: 0 >> page_cache_hdr[11]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 218a1d0 >> pg_hit_count: 0 >> page_cache_hdr[12]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 218b1d0 >> pg_hit_count: 0 >> page_cache_hdr[13]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 218c1d0 >> pg_hit_count: 0 >> page_cache_hdr[14]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 218d1d0 >> pg_hit_count: 0 >> page_cache_hdr[15]: >> pg_flags: 0 () >> pg_addr: 0 >> pg_bufptr: 218e1d0 >> pg_hit_count: 0 >> >> page_cache_buf: 217f1d0 >> evict_index: 0 >> evictions: 0 >> accesses: 0 >> cached_reads: 0 >> valid_pages: 212a3b0 >> readmem: read_diskdump() >> KASLR: >> _stext from vmlinux.debuginfo: ffffffff81000000 >> _stext from vmcoreinfo: ffffffffb4800000 >> relocate: 33800000 (824MB) >> crash: pv_init_ops exists: ARCH_PVOPS >> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000 gdb >> vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software >> Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later >> <http://gnu.org/licenses/gpl.html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-unknown-linux-gnu"... >> GETBUF(328 -> 0) >> GETBUF(1500 -> 1) >> >> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol >> values >> >> FREEBUF(1) >> FREEBUF(0) >> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q), >> d8b948> >> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8> >> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page: >> 3ecdf000 >> GETBUF(328 -> 0) >> FREEBUF(0) >> GETBUF(1024 -> 0) >> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120> >> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024> >> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c >> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type: "possible" >> WARNING: cannot read cpu_possible_map >> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120> >> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544> >> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b >> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type: "present" >> WARNING: cannot read cpu_present_map >> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120> >> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024> >> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c >> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type: "online" >> WARNING: cannot read cpu_online_map >> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120> >> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024> >> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b >> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type: "active" >> WARNING: cannot read cpu_active_map >> FREEBUF(0) >> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE), >> 7ffd3655d640> >> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8> >> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page: >> 3ef1a000 >> GETBUF(328 -> 0) >> FREEBUF(0) >> GETBUF(328 -> 0) >> FREEBUF(0) >> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8, >> (ROE), 7ffd3655d5f0> >> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8> >> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad >> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type: "shadow_timekeeper xtime_sec" >> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970 >> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE), d71b7c> >> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390> >> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page: >> 3ee21000 >> utsname: >> sysname: >> nodename: >> release: >> version: >> machine: >> domainname: >> base kernel version: 0.0.0 >> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q), >> 7ffd3655af30> >> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8> >> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601 >> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type: "accessible check" >> crash: vmlinux.debuginfo and vmcore do not match! >> >> Usage: >> >> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) >> crash [OPTION]... [NAMELIST] (live system form) >> >> Enter "crash -h" for details. >> > > -- > Crash-utility mailing list > Crash-utility@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/crash-utility > > > -- > Crash-utility mailing list > Crash-utility@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/crash-utility > -- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility
