Good point, enough the –kaslr=auto option worked well. Same when I passed
--kaslr=0x8000000
root@instance-2:~# crash --kaslr=auto vmlinux-17162.336.25 /proc/kcore
crash 8.0.4
Copyright (C) 2002-2022
Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010
IBM Corporation
Copyright (C) 1999-2006
Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012
Fujitsu Limited
Copyright (C) 2006, 2007
VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022
NEC Corporation
Copyright (C) 1999, 2002, 2007
Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002
Mission Critical Linux, Inc.
Copyright (C) 2015, 2021
VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.
Enter "help copying" to see the conditions.
This program has absolutely no warranty.
Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: vmlinux-17162.336.25
[TAINTED]
DUMPFILE: /proc/kcore
CPUS: 2
DATE: Wed Nov 22 06:37:56 UTC 2023
UPTIME: 19:15:54
LOAD AVERAGE: 0.15, 0.03, 0.01
TASKS: 132
NODENAME: instance-2
RELEASE: 5.15.133+
VERSION: #1 SMP Sat Nov 11 11:15:28 UTC 2023
MACHINE: x86_64
(2249 Mhz)
MEMORY: 4 GB
PID: 160180
COMMAND: "crash"
TASK: ffff8ec242ec53c0
[THREAD_INFO: ffff8ec242ec53c0]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash> ps
PID
PPID CPU
TASK
ST %MEM
VSZ
RSS COMM
0
0
0
ffffffff8a616540
RU 0.0
0
0 [swapper/0]
>
0
0 1
ffff8ec240276480
RU 0.0
0
0 [swapper/1]
1
0
1
ffff8ec24025c300
IN 0.2
96020
9660 systemd
2
0
0
ffff8ec240258000
IN 0.0
0
0 [kthreadd]
3
2
0
ffff8ec24025e480
ID 0.0
0
0 [rcu_gp]
4
2
0
ffff8ec24025a180
ID 0.0
0
0 [rcu_par_gp]
5
2
0
ffff8ec24025b240
ID 0.0
0
0 [slub_flushwq]
From:
HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab@xxxxxxx>
Date: Wednesday, November 22, 2023 at 8:36 AM
To: Matt Suiche <matt.suiche@xxxxxxxxxxxxxxxxxxx>, devel@xxxxxxxxxxxxxxxxxxxxxxxxxxx <devel@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Subject: EXTERNAL SENDER Re: Google Container OS and crash 8.0.4
Hi Matt,
Thank you for trying the latest.
> SYMBOL(_stext)=ffffffff89000000
> KERNELOFFSET=8000000
> <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q),
5642aae35c08>
$ curl -O
https://can01.safelinks.protection.outlook.com/?url="">
$ nm vmlinux | grep -e ' _stext' -e ' page_offset_base'
ffffffff81000000 T _stext
ffffffff82239750 R page_offset_base
To me, it looks like KASLR detection doesn't work. The randomized
offset of the page_offset_base should be 0xffffffff82239750 + 0x8000000
= 0xffffffff8a239750, but crash is trying to read 0xffffffff82239750.
We need to look into why it doesn't work, firstly does this option work?
If this works, I think it will be a clue.
# crash --kaslr=auto vmlinux /proc/kcore
or
# crash --kaslr=<KERNELOFFSET value> vmlinux /proc/kcore
i.e. --kaslr=8000000 during that system session.
(this will vary after system reboot)
Thanks,
Kazu
On 2023/11/21 23:21, Matt Suiche wrote:
> Dear,
>
> I tried to use crash 8.0.4 on Google Container OS (17162.336.25) but for some reason there is resistance.
>
> Step to reproduce:
>
> 1. Create a Virtual Machine in Google Cloud using Google Container OS as a base image
> 2. Run “toolkit”
> 3. Download the vmlinux symbols for the current base image
> * curl
https://can01.safelinks.protection.outlook.com/?url="">
> symbols/vmlinux-$container_host_build_id
> 4. Run crash on /proc/kcore
>
> Thanks,
>
> Logs:
>
> root@instance-2:~# crash /proc/kcore vmlinux-17162.336.25 -d 99
>
>
>
> crash 8.0.4
>
> Copyright (C) 2002-2022 Red Hat, Inc.
>
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
>
> Copyright (C) 1999-2006 Hewlett-Packard Co
>
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
>
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>
> Copyright (C) 2005, 2011, 2020-2022 NEC Corporation
>
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
>
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>
> Copyright (C) 2015, 2021 VMware, Inc.
>
> This program is free software, covered by the GNU General Public License,
>
> and you are welcome to change it and/or distribute copies of it under
>
> certain conditions. Enter "help copying" to see the conditions.
>
> This program has absolutely no warranty. Enter "help warranty" for details.
>
>
>
> get_live_memory_source: /proc/kcore
>
> proc_kcore_data:
>
> flags: 500 (KCORE_LOCAL|KCORE_ELF64)
>
> segments: 12
>
> elf_header: 5642ab6d3f40
>
> header_size: 8636
>
> notes64: 5642ab6d3f80
>
> load64: 5642ab6d3fb8
>
> notes32: 0
>
> load32: 0
>
> vmcoreinfo: 0
>
> size_vmcoreinfo: 0
>
>
>
> Elf64_Phdr:
>
> p_type: 4 (PT_NOTE)
>
> p_flags: 0
>
> p_offset: 318
>
> p_vaddr: 0
>
> p_paddr: 0
>
> p_filesz: 7844
>
> p_memsz: 0
>
> p_align: 0
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7fff89003000
>
> p_vaddr: ffffffff89000000
>
> p_paddr: 13a000000
>
> p_filesz: 35831808
>
> p_memsz: 35831808
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 130900003000
>
> p_vaddr: ffff930900000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 35184372088831
>
> p_memsz: 35184372088831
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7fffc0003000
>
> p_vaddr: ffffffffc0000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 1056964608
>
> p_memsz: 1056964608
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec140004000
>
> p_vaddr: ffff8ec140001000
>
> p_paddr: 1000
>
> p_filesz: 344064
>
> p_memsz: 344064
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c0003000
>
> p_vaddr: fffffbe8c0000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 8192
>
> p_memsz: 8192
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec140063000
>
> p_vaddr: ffff8ec140060000
>
> p_paddr: 60000
>
> p_filesz: 229376
>
> p_memsz: 229376
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec140103000
>
> p_vaddr: ffff8ec140100000
>
> p_paddr: 100000
>
> p_filesz: 3212759040
>
> p_memsz: 3212759040
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c0007000
>
> p_vaddr: fffffbe8c0004000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 50200576
>
> p_memsz: 50200576
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec1ffc02000
>
> p_vaddr: ffff8ec1ffbff000
>
> p_paddr: bfbff000
>
> p_filesz: 4067328
>
> p_memsz: 4067328
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c2ff2000
>
> p_vaddr: fffffbe8c2fef000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 69632
>
> p_memsz: 69632
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec240003000
>
> p_vaddr: ffff8ec240000000
>
> p_paddr: 100000000
>
> p_filesz: 1073741824
>
> p_memsz: 1073741824
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c4003000
>
> p_vaddr: fffffbe8c4000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 16777216
>
> p_memsz: 16777216
>
> p_align: 4096
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 5 ("CORE")
>
> n_descsz: 336
>
> n_type: 1 (NT_PRSTATUS)
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 5 ("CORE")
>
> n_descsz: 136
>
> n_type: 3 (NT_PRPSINFO)
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 5 ("CORE")
>
> n_descsz: 4288
>
> n_type: 4 (NT_TASKSTRUCT)
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 11 ("VMCOREINFO")
>
> n_descsz: 3000
>
> n_type: 0 (unknown)
>
>
>
> OSRELEASE=5.15.133+
>
> BUILD-ID=f16c9f1b53617d7b151c4d18d79c6ccbb44ea6d6
>
> PAGESIZE=4096
>
> SYMBOL(init_uts_ns)=ffffffff8a615698
>
> OFFSET(uts_namespace.name)=0
>
> SYMBOL(node_online_map)=ffffffff8a85d638
>
> SYMBOL(swapper_pg_dir)=ffffffff8a60c000
>
> SYMBOL(_stext)=ffffffff89000000
>
> SYMBOL(vmap_area_list)=ffffffff8a774208
>
> SYMBOL(mem_section)=ffff8ec27fff8000
>
> LENGTH(mem_section)=2048
>
> SIZE(mem_section)=16
>
> OFFSET(mem_section.section_mem_map)=0
>
> NUMBER(SECTION_SIZE_BITS)=27
>
> NUMBER(MAX_PHYSMEM_BITS)=46
>
> SIZE(page)=64
>
> SIZE(pglist_data)=15616
>
> SIZE(zone)=1664
>
> SIZE(free_area)=104
>
> SIZE(list_head)=16
>
> SIZE(nodemask_t)=8
>
> OFFSET(page.flags)=0
>
> OFFSET(page._refcount)=52
>
> OFFSET(page.mapping)=24
>
> OFFSET(page.lru)=8
>
> OFFSET(page._mapcount)=48
>
> OFFSET(page.private)=40
>
> OFFSET(page.compound_dtor)=16
>
> OFFSET(page.compound_order)=17
>
> OFFSET(page.compound_head)=8
>
> OFFSET(pglist_data.node_zones)=0
>
> OFFSET(pglist_data.nr_zones)=14880
>
> OFFSET(pglist_data.node_start_pfn)=14888
>
> OFFSET(pglist_data.node_spanned_pages)=14904
>
> OFFSET(pglist_data.node_id)=14912
>
> OFFSET(zone.free_area)=192
>
> OFFSET(zone.vm_stat)=1472
>
> OFFSET(zone.spanned_pages)=128
>
> OFFSET(free_area.free_list)=0
>
> OFFSET(list_head.next)=0
>
> OFFSET(list_head.prev)=8
>
> OFFSET(vmap_area.va_start)=0
>
> OFFSET(vmap_area.list)=40
>
> LENGTH(zone.free_area)=11
>
> SYMBOL(prb)=ffffffff8a662318
>
> SYMBOL(printk_rb_static)=ffffffff8a662320
>
> SYMBOL(clear_seq)=ffffffff8ad8c0d8
>
> SIZE(printk_ringbuffer)=80
>
> OFFSET(printk_ringbuffer.desc_ring)=0
>
> OFFSET(printk_ringbuffer.text_data_ring)=40
>
> OFFSET(printk_ringbuffer.fail)=72
>
> SIZE(prb_desc_ring)=40
>
> OFFSET(prb_desc_ring.count_bits)=0
>
> OFFSET(prb_desc_ring.descs)=8
>
> OFFSET(prb_desc_ring.infos)=16
>
> OFFSET(prb_desc_ring.head_id)=24
>
> OFFSET(prb_desc_ring.tail_id)=32
>
> SIZE(prb_desc)=24
>
> OFFSET(prb_desc.state_var)=0
>
> OFFSET(prb_desc.text_blk_lpos)=8
>
> SIZE(prb_data_blk_lpos)=16
>
> OFFSET(prb_data_blk_lpos.begin)=0
>
> OFFSET(prb_data_blk_lpos.next)=8
>
> SIZE(printk_info)=88
>
> OFFSET(printk_info.seq)=0
>
> OFFSET(printk_info.ts_nsec)=8
>
> OFFSET(printk_info.text_len)=16
>
> OFFSET(printk_info.caller_id)=20
>
> OFFSET(printk_info.dev_info)=24
>
> SIZE(dev_printk_info)=64
>
> OFFSET(dev_printk_info.subsystem)=0
>
> LENGTH(printk_info_subsystem)=16
>
> OFFSET(dev_printk_info.device)=16
>
> LENGTH(printk_info_device)=48
>
> SIZE(prb_data_ring)=32
>
> OFFSET(prb_data_ring.size_bits)=0
>
> OFFSET(prb_data_ring.data)=8
>
> OFFSET(prb_data_ring.head_lpos)=16
>
> OFFSET(prb_data_ring.tail_lpos)=24
>
> SIZE(atomic_long_t)=8
>
> OFFSET(atomic_long_t.counter)=0
>
> SIZE(latched_seq)=24
>
> OFFSET(latched_seq.val)=8
>
> LENGTH(free_area.free_list)=6
>
> NUMBER(NR_FREE_PAGES)=0
>
> NUMBER(PG_lru)=4
>
> NUMBER(PG_private)=13
>
> NUMBER(PG_swapcache)=10
>
> NUMBER(PG_swapbacked)=19
>
> NUMBER(PG_slab)=9
>
> NUMBER(PG_hwpoison)=23
>
> NUMBER(PG_head_mask)=65536
>
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
>
> NUMBER(HUGETLB_PAGE_DTOR)=2
>
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
>
> NUMBER(phys_base)=5117050880
>
> SYMBOL(init_top_pgt)=ffffffff8a60c000
>
> NUMBER(pgtable_l5_enabled)=0
>
> SYMBOL(node_data)=ffffffff8a85c5d0
>
> LENGTH(node_data)=64
>
> KERNELOFFSET=8000000
>
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>
> NUMBER(sme_mask)=0
>
>
>
> /proc/version:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD
14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023
>
> vmlinux-17162.336.25:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD
14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023
>
> readmem: read_proc_kcore() -> /proc/kcore
>
> crash: pv_ops exists: ARCH_PVOPS
>
> VMCOREINFO: NUMBER(phys_base): 5117050880 -> 131000000
>
> gdb vmlinux-17162.336.25
>
> GNU gdb (GDB) 10.2
>
> Copyright (C) 2021 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later
https://can01.safelinks.protection.outlook.com/?url="">
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law.
>
> Type "show copying" and "show warranty" for details.
>
> This GDB was configured as "x86_64-pc-linux-gnu".
>
> Type "show configuration" for configuration details.
>
> Find the GDB manual and other documentation resources online at:
>
> https://can01.safelinks.protection.outlook.com/?url="">.
>
>
>
> For help, type "help".
>
> Type "apropos word" to search for commands related to "word"...
>
> GETBUF(344 -> 0)
>
> GETBUF(1500 -> 1)
>
>
>
> FREEBUF(1)
>
> FREEBUF(0)
>
> <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q), 5642aae35c08>
>
> <read_proc_kcore: addr: ffffffff82239750 paddr: 133239750 cnt: 8>
>
> crash: seek error: kernel virtual address: ffffffff82239750 type: "page_offset_base"
>
>
>
> root@instance-2:~# env
>
> container_host_version_id=101
>
> PWD=/root
>
> LOGNAME=root
>
> container=systemd-nspawn
>
> HOME=/root
>
> TERM=xterm-256color
>
> USER=root
>
> NOTIFY_SOCKET=/run/host/notify
>
> SHLVL=1
>
> container_host_id=cos
>
> container_host_build_id=17162.336.25
>
> PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>
> container_uuid=d8282d15-c11a-416b-9371-94db01a7ca15
>
> _=/usr/bin/env
>
> OLDPWD=/
>
>
> This email including any attachments may contain confidential material for the sole use of the intended recipient. If you are not the intended recipient please immediately notify the sender by reply email, permanently delete this message and do not forward
it or any part of it to anyone else.
>
This email including any attachments may contain confidential material for the sole use of the intended recipient. If you are not the intended recipient please immediately notify the sender by reply email, permanently delete this message and do not forward
it or any part of it to anyone else.
