Hi Tao,
Thanks for your great help, after integrating the patch, the crash tool is working in my environment.
Currently, with -S and --kaslr=auto, the crash can enter the CLI normally, and symbols display correct. Thanks!
Regaring without -S reporting "crash: cannot resolve "init_task_union", I am still invesgating, seems related the symtab
in my own kernel image, once have some new finding, I will update the result in this thread.
Thanks!
Br,
Xiangyu
在 2024年12月13日 星期五 上午11:20:55 [GMT+8]， Tao Liu<ltao@xxxxxxxxxx> 寫道：
Hi Xiangyu,
On Thu, Dec 12, 2024 at 4:15 PM Xiangyu Chen <xiangyu.chen@xxxxxxx> wrote:
>
> Hi Tao,
>
> Thanks for your reply,
>
Let me try to answer your questions.
> 1) Try to drop the "-S" of crash cmdline to see if it works.
> I have tried without -S, it was broken by another issue, the crash reported an error :
>
> _note_9: st_name: 1692 st_value: ffffffffc04d747c st_shndx: 22 st_info: ?
> __this_module: st_name: 1700 st_value: ffffffffc04d8380 st_shndx: 29 st_info: ?
> cleanup_module: st_name: 1714 st_value: ffffffffc04d6c48 st_shndx: 7 st_info: T
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffffab2df068, KVADDR, "nr_threads", 4, (FOE), 55f06caa495c>
> <read_proc_kcore: addr: ffffffffab2df068 paddr: 11cedf068 cnt: 4>
> crash: per_cpu_symbol_search(per_cpu__runqueues): NULL
> crash: cannot determine idle task addresses from init_tasks[] or runqueues[]
>
>
> crash: cannot resolve "init_task_union"
Without '-S', crash will try to find the symbols within vmlinux, but
somehow it failed to find the symbols as specified in
tasks.c:get_idle_threads(). So it fails down to reach
"init_task_union", which is a very old symbol and no longer exists in
kernels nowadays. I cannot tell why the symbols are missing in your
kernel, maybe due to your gcc or kernel config or else.
>
>
> 2) If not, then please try to disable the kernel kaslr at bootup by
> appending "nokaslr" to kernel cmdline, and invoke crash as your
> original way.
> After append the "nokaslr" to kernel command line, the crash without -S still report "crash: cannot resolve "init_task_union"
> But with "-S" parameter, crash can enter the CLI as normal.
>
Currently the "-S" option is problematic and it doesn't deal with
kaslr issue. So if you use "-S" with kaslr enabled, it will fail with
no doubt. So when you disabled kaslr and use "-S" at the same time,
this can work as expected.
Please check the attachments, in which I drafted a patch to fix the
kaslr issue for "-S". This should work when you use "-S" with kaslr
enabled by your kernel. Please see if it can work for you.
Thanks,
Tao Liu
>
> So, regarding the behavior, when without -S, the crash report "crash: cannot resolve "init_task_union"", this should
> not related to kaslr, right? If so, is that means missing kernel configuration/patch or gdb needs a special version?
> The current kernel is 6.1 stable, gdb version is 10.2.
>
>
>
> Thanks,
>
> Br,
> Xiangyu
>
>
> 在 2024年12月12日 星期四 上午07:46:05 [GMT+8]， Tao Liu<ltao@xxxxxxxxxx> 寫道：
>
> Hi xiangyu,
>
> On Fri, Dec 6, 2024 at 9:25 PM <xiangyu.chen@xxxxxxx> wrote:
> >
> > Hi Crash-utility folks,
> >
> > Recently, I am using crash to debug a live system.
> > When I execute the crash, it cannot enter the crash cmd shell, it was broken by crash seek error.
> > My kernel turned on the CONFIG_RANDOMIZE_BASE, and I added the kaslr=auto parameter to crash.
> > If I disable the CONFIG_RANDOMIZE_BASE, the issue has gone.
> > Could you please help to check if any config/parameter I was missing? Thanks.
> >
> > The following is my system information:
> > crash version: 8.0.6
> > kernel version: 6.1.118
> > Host system: qemux86-64
> >
> > Here is the full log with -d:
> > root@intel-x86-64:/boot# crash -S /boot/vmlinux-6.1.118-yocto-standard /proc/kcore -d10 --kaslr=auto
> >
> 1) Try to drop the "-S" of crash cmdline to see if it works.
> 2) If not, then please try to disable the kernel kaslr at bootup by
> appending "nokaslr" to kernel cmdline, and invoke crash as your
> original way.
>
> Hope this helps.
>
> Thanks,
> Tao Liu
>
> > crash 8.0.6
> > Copyright (C) 2002-2025 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > Copyright (C) 2005, 2011, 2020-2024 NEC Corporation
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > Copyright (C) 2015, 2021 VMware, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions. Enter "help copying" to see the conditions.
> > This program has absolutely no warranty. Enter "help warranty" for details.
> >
> > get_live_memory_source: /proc/kcore
> > proc_kcore_data:
> > flags: 500 (KCORE_LOCAL|KCORE_ELF64)
> > segments: 10
> > elf_header: 56471f1eb9c0
> > header_size: 12324
> > notes64: 56471f1eba00
> > load64: 56471f1eba38
> > notes32: 0
> > load32: 0
> > vmcoreinfo: 0
> > size_vmcoreinfo: 0
> >
> > Elf64_Phdr:
> > p_type: 4 (PT_NOTE)
> > p_flags: 0
> > p_offset: 2a8
> > p_vaddr: 0
> > p_paddr: 0
> > p_filesz: 11644
> > p_memsz: 0
> > p_align: 0
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 7fffaa404000
> > p_vaddr: ffffffffaa400000
> > p_paddr: b8600000
> > p_filesz: 33751040
> > p_memsz: 33751040
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 286600004000
> > p_vaddr: ffffa86600000000
> > p_paddr: ffffffffffffffff
> > p_filesz: 35184372088831
> > p_memsz: 35184372088831
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 7fffc0004000
> > p_vaddr: ffffffffc0000000
> > p_paddr: ffffffffffffffff
> > p_filesz: 1056964608
> > p_memsz: 1056964608
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 126ac0005000
> > p_vaddr: ffff926ac0001000
> > p_paddr: 1000
> > p_filesz: 647168
> > p_memsz: 647168
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 4d8880004000
> > p_vaddr: ffffcd8880000000
> > p_paddr: ffffffffffffffff
> > p_filesz: 12288
> > p_memsz: 12288
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 126ac0104000
> > p_vaddr: ffff926ac0100000
> > p_paddr: 100000
> > p_filesz: 12902400
> > p_memsz: 12902400
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 4d8880008000
> > p_vaddr: ffffcd8880004000
> > p_paddr: ffffffffffffffff
> > p_filesz: 204800
> > p_memsz: 204800
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 126ac0d53000
> > p_vaddr: ffff926ac0d4f000
> > p_paddr: d4f000
> > p_filesz: 3207118848
> > p_memsz: 3207118848
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 126bc0004000
> > p_vaddr: ffff926bc0000000
> > p_paddr: 100000000
> > p_filesz: 4294967296
> > p_memsz: 4294967296
> > p_align: 4096
> >
> > Elf64_Phdr:
> > p_type: 1 (PT_LOAD)
> > p_flags: 7
> > p_offset: 4d8884004000
> > p_vaddr: ffffcd8884000000
> > p_paddr: ffffffffffffffff
> > p_filesz: 67108864
> > p_memsz: 67108864
> > p_align: 4096
> >
> > Elf64_Nhdr:
> > n_namesz: 5 ("CORE")
> > n_descsz: 336
> > n_type: 1 (NT_PRSTATUS)
> >
> > Elf64_Nhdr:
> > n_namesz: 5 ("CORE")
> > n_descsz: 136
> > n_type: 3 (NT_PRPSINFO)
> >
> > Elf64_Nhdr:
> > n_namesz: 5 ("CORE")
> > n_descsz: 7808
> > n_type: 4 (NT_TASKSTRUCT)
> >
> > Elf64_Nhdr:
> > n_namesz: 11 ("VMCOREINFO")
> > n_descsz: 3279
> > n_type: 0 (unknown)
> >
> > OSRELEASE=6.1.118-yocto-standard
> > BUILD-ID=1400000003000000474e5500a343709706409b9c
> > PAGESIZE=4096
> > SYMBOL(init_uts_ns)=ffffffffac037a80
> > OFFSET(uts_namespace.name)=0
> > SYMBOL(node_online_map)=ffffffffac06d3e8
> > SYMBOL(swapper_pg_dir)=ffffffffabe0a000
> > SYMBOL(_stext)=ffffffffaa400000
> > SYMBOL(vmap_area_list)=ffffffffabf12fb0
> > SYMBOL(mem_section)=ffff926cbfff5000
> > LENGTH(mem_section)=2048
> > SIZE(mem_section)=16
> > OFFSET(mem_section.section_mem_map)=0
> > NUMBER(SECTION_SIZE_BITS)=27
> > NUMBER(MAX_PHYSMEM_BITS)=46
> > SIZE(page)=64
> > SIZE(pglist_data)=15616
> > SIZE(zone)=1600
> > SIZE(free_area)=104
> > SIZE(list_head)=16
> > SIZE(nodemask_t)=8
> > OFFSET(page.flags)=0
> > OFFSET(page._refcount)=52
> > OFFSET(page.mapping)=24
> > OFFSET(page.lru)=8
> > OFFSET(page._mapcount)=48
> > OFFSET(page.private)=40
> > OFFSET(page.compound_dtor)=16
> > OFFSET(page.compound_order)=17
> > OFFSET(page.compound_head)=8
> > OFFSET(pglist_data.node_zones)=0
> > OFFSET(pglist_data.nr_zones)=14624
> > OFFSET(pglist_data.node_start_pfn)=14632
> > OFFSET(pglist_data.node_spanned_pages)=14648
> > OFFSET(pglist_data.node_id)=14656
> > OFFSET(zone.free_area)=192
> > OFFSET(zone.vm_stat)=1472
> > OFFSET(zone.spanned_pages)=136
> > OFFSET(free_area.free_list)=0
> > OFFSET(list_head.next)=0
> > OFFSET(list_head.prev)=8
> > OFFSET(vmap_area.va_start)=0
> > OFFSET(vmap_area.list)=40
> > LENGTH(zone.free_area)=11
> > SYMBOL(prb)=ffffffffabe647d8
> > SYMBOL(printk_rb_static)=ffffffffabe647e0
> > SYMBOL(clear_seq)=ffffffffac307f20
> > SIZE(printk_ringbuffer)=88
> > OFFSET(printk_ringbuffer.desc_ring)=0
> > OFFSET(printk_ringbuffer.text_data_ring)=48
> > OFFSET(printk_ringbuffer.fail)=80
> > SIZE(prb_desc_ring)=48
> > OFFSET(prb_desc_ring.count_bits)=0
> > OFFSET(prb_desc_ring.descs)=8
> > OFFSET(prb_desc_ring.infos)=16
> > OFFSET(prb_desc_ring.head_id)=24
> > OFFSET(prb_desc_ring.tail_id)=32
> > SIZE(prb_desc)=24
> > OFFSET(prb_desc.state_var)=0
> > OFFSET(prb_desc.text_blk_lpos)=8
> > SIZE(prb_data_blk_lpos)=16
> > OFFSET(prb_data_blk_lpos.begin)=0
> > OFFSET(prb_data_blk_lpos.next)=8
> > SIZE(printk_info)=88
> > OFFSET(printk_info.seq)=0
> > OFFSET(printk_info.ts_nsec)=8
> > OFFSET(printk_info.text_len)=16
> > OFFSET(printk_info.caller_id)=20
> > OFFSET(printk_info.dev_info)=24
> > SIZE(dev_printk_info)=64
> > OFFSET(dev_printk_info.subsystem)=0
> > LENGTH(printk_info_subsystem)=16
> > OFFSET(dev_printk_info.device)=16
> > LENGTH(printk_info_device)=48
> > SIZE(prb_data_ring)=32
> > OFFSET(prb_data_ring.size_bits)=0
> > OFFSET(prb_data_ring.data)=8
> > OFFSET(prb_data_ring.head_lpos)=16
> > OFFSET(prb_data_ring.tail_lpos)=24
> > SIZE(atomic_long_t)=8
> > OFFSET(atomic_long_t.counter)=0
> > SIZE(latched_seq)=24
> > OFFSET(latched_seq.val)=8
> > LENGTH(free_area.free_list)=6
> > NUMBER(NR_FREE_PAGES)=0
> > NUMBER(PG_lru)=4
> > NUMBER(PG_private)=13
> > NUMBER(PG_swapcache)=10
> > NUMBER(PG_swapbacked)=19
> > NUMBER(PG_slab)=9
> > NUMBER(PG_hwpoison)=23
> > NUMBER(PG_head_mask)=65536
> > NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> > NUMBER(HUGETLB_PAGE_DTOR)=2
> > NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> > SYMBOL(kallsyms_names)=ffffffffaba0f7d8
> > SYMBOL(kallsyms_num_syms)=ffffffffaba0f7d0
> > SYMBOL(kallsyms_token_table)=ffffffffabbf5400
> > SYMBOL(kallsyms_token_index)=ffffffffabbf5788
> > SYMBOL(kallsyms_offsets)=ffffffffab9984f8
> > SYMBOL(kallsyms_relative_base)=ffffffffaba0f7c8
> > NUMBER(phys_base)=2384461824
> > SYMBOL(init_top_pgt)=ffffffffabe0a000
> > NUMBER(pgtable_l5_enabled)=0
> > SYMBOL(node_data)=ffffffffac06c3a0
> > LENGTH(node_data)=64
> > KERNELOFFSET=29400000
> > NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> > NUMBER(sme_mask)=0
> >
> > readmem: read_proc_kcore() -> /proc/kcore
> > VMCOREINFO: NUMBER(phys_base): 2384461824 -> 8e200000
> > gdb /boot/vmlinux-6.1.118-yocto-standard
> > GNU gdb (GDB) 10.2
> > Copyright (C) 2021 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.
> > Type "show copying" and "show warranty" for details.
> > This GDB was configured as "x86_64-wrs-linux".
> > Type "show configuration" for configuration details.
> > Find the GDB manual and other documentation resources online at:
> > <http://www.gnu.org/software/gdb/documentation/>.
> >
> > For help, type "help".
> > Type "apropos word" to search for commands related to "word"...
> > GETBUF(344 -> 0)
> > GETBUF(1500 -> 1)
> >
> > FREEBUF(1)
> > FREEBUF(0)
> > <readmem: ffffffff8292c360, KVADDR, "page_offset_base", 8, (FOE|Q), 56471ef799c8>
> > <read_proc_kcore: addr: ffffffff8292c360 paddr: 90b2c360 cnt: 8>
> > crash: seek error: kernel virtual address: ffffffff8292c360 type: "page_offset_base"
> > --
> > Crash-utility mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxxxxxx
> > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxxxxxx
> > https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> > Contribution Guidelines: https://github.com/crash-utility/crash/wiki
>
--
Crash-utility mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxxxxxx
https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
Contribution Guidelines: https://github.com/crash-utility/crash/wiki
