Hello MIPS Linux community,
We are trying to use Linux kernel vmcore feature to capture the kernel
crash information on Cavium octeon board. Particularly on octeon2
(68xx) board. Could you please help in finding a way to debug this.
Issue is, when the second kernel is getting booted, at the very
initial stage of kernel boot we are getting "Kernel unaligned
instruction access[#1]:" error and the kernel is crashing.
Second kernel code, which is getting executed before kernel crash.
====
FEXPORT(octeon_hotplug_entry)
move a0, zero
move a0, zero
move a2, zero
move a3, zero
7:
#endif /* CONFIG_HOTPLUG_CPU */
mfc0 v0, CP0_STATUS
/* Force 64-bit addressing enabled */
ori v0, v0, (ST0_UX | ST0_SX | ST0_KX)
/* Clear NMI and SR as they are sometimes restored and 0 -> 1
* transitions are not allowed
*/
li v1, ~(ST0_NMI | ST0_SR)
and v0, v1
mtc0 v0, CP0_STATUS
# Clear the TLB.
mfc0 v0, $16, 1 # Config1
dsrl v0, v0, 25
andi v0, v0, 0x3f
mfc0 v1, $16, 3 # Config3
bgez v1, 1f
mfc0 v1, $16, 4 # Config4
andi v1, 0x7f
dsll v1, 6
or v0, v0, v1
1: # Number of TLBs in v0
dmtc0 zero, $2, 0 # EntryLo0
dmtc0 zero, $3, 0 # EntryLo1
dmtc0 zero, $5, 0 # PageMask
tlbwi # Make it a 'normal' sized page
daddiu t0, t0, 8192
b 10b
1:
mtc0 v0, $0, 0 # Index
tlbwi
.set noreorder
bne v0, zero, 10b
addiu v0, v0, -1
.set reorder
mtc0 zero, $0, 0 # Index
dmtc0 zero, $10, 0 # EntryHi
dla v0, continue_in_mapped_space
jr v0
continue_in_mapped_space:
mfc0 v1, CP0_PRID_REG
...
=====
Here is the objdump of continue_in_mapped_space
ffffffff0be3f7b4 <continue_in_mapped_space>:
ffffffff0be3f7b4: 40037800 mfc0 v1,$15
ffffffff0be3f7b8: 3063ff00 andi v1,v1,0xff00
ffffffff0be3f7bc: 34029500 li v0,0x9500
ffffffff0be3f7c0: 00621823 subu v1,v1,v0
ffffffff0be3f7c4: 240e0002 li t2,2
ffffffff0be3f7c8: 04600002 bltz v1,ffffffff0be3f7d4
<continue_in_mapped_space+0x20>
ffffffff0be3f7cc: 00000000 nop
ffffffff0be3f7d0: 25ce0001 addiu t2,t2,1
ffffffff0be3f7d4: 40225807 dmfc0 v0,$11,7
Kernel unaligned instruction access[#1]: issue is happening at start
of continue_in_mapped_space function.
GCC used for compilation.
./mips64-linux-gnu-gcc --version
mips64-linux-gnu-gcc 4.9.4
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Kernel is compiled with option -mabi=64 .
Second Kernel crash logs are attached.
--
Guruswamy Basavaiah
echo c > /proc/sysrq-trigger ;5F
[86365.400562] sysrq: Trigger a crash
[86365.403988] CPU 0 Unable to handle kernel paging request at virtual address 0000000000000000, epc == ffffffff80bcfe5c, ra == ffffffff80bd04b4
[86365.416734] Oops[#1]:
[86365.419019] CPU: 0 PID: 817 Comm: bash Tainted: G O 4.4.245-octeon-distro.git-v2.109-1-rc #1
[86365.428586] task: 80000003fa59de80 ti: 80000003fc934000 task.ti: 80000003fc934000
[86365.436067] $ 0 : 0000000000000000 ffffffff80bd04b4 0000000000000001 ffffffff81ba0000
[86365.444134] $ 4 : 0000000000000063 80000000a5130558 80000000a5132b68 0000000000000005
[86365.452202] $ 8 : ffffffffc027e260 0000000000000221 ffffffff81bf0000 ffffffff810d0000
[86365.460270] $12 : 0000000000000000 0000000000000000 0000000000000220 0000000000000000
[86365.468340] $16 : ffffffff8111ee80 0000000000000063 0000000000000004 0000000000000000
[86365.476408] $20 : ffffffff810d0000 ffffffff8111eb60 ffffffff81120000 000000ffffb69cd4
[86365.484480] $24 : 0000000000000006 ffffffff80bddc68
[86365.492547] $28 : 80000003fc934000 80000003fc937cb0 000000ffea89e6c8 ffffffff80bd04b4
[86365.500614] Hi : 000000000061cb2f
[86365.504190] Lo : df3b645a1d20297a
[86365.507771] epc : ffffffff80bcfe5c sysrq_handle_crash+0x1c/0x28
[86365.513867] ra : ffffffff80bd04b4 __handle_sysrq+0x13c/0x1d8
[86365.519787] Status: 10008ce3 KX SX UX KERNEL EXL IE
[86365.524840] Cause : 8080000c (ExcCode 03)
[86365.528848] BadVA : 0000000000000000
[86365.532422] PrId : 000d9101 (Cavium Octeon II)
[86365.536952] Modules linked in: ecc_driver_edac(O) ecc_driver_core(O) netconsole octeon_edac_pci octeon_edac_l2c octeon_edac_pc octeon_edac_lmc edac_core mtdoops isofs ehci_platform ehci_hcd physmap autofs4 nfsv3 nfs_acl ipmi_watchdog 8021q garp stp mrp llc ipmi_si ipmi_devintf ipmi_msghandler nfs lockd grace sunrpc octeon_ethernet ethernet_mem igb hwmon i2c_algo_bit e1000e ptp pps_core
[86365.571600] Process bash (pid: 817, threadinfo=80000003fc934000, task=80000003fa59de80, tls=000000ffea983f60)
[86365.581512] Stack : 80000000058a8000 80000003fa77b2c0 fffffffffffffffb 0000000000000002
0000000120f7cc00 0000000000000002 000000ffea7e8050 000000012012b158
000000ffffb69cd4 ffffffff80bd05d8 0000000000000002 80000000058a8022
0000000000000000 ffffffff80a4af58 80000000efd9dd78 80000003fdc9de00
80000003fc937e00 ffffffff809e1c94 0000000400000000 0000000000000000
00000000f00003ad 0000808000000001 0000000000000000 0000000000000000
0000000000000000 0000000000000000 605c5470219fef34 605c5470219fef34
605c5470219fef34 80000003fdc9de00 80000003fc937e00 80000003fdc9de00
80000003fc937e00 ffffffff809e2634 000000ffea89e5e0 ffffffff809e838c
80000003fdc9de00 fffffffffdc9de00 0000000000000002 0000000120f7cc00
...
[86365.647167] Call Trace:
[86365.649616] [<ffffffff80bcfe5c>] sysrq_handle_crash+0x1c/0x28
[86365.655367] [<ffffffff80bd04b4>] __handle_sysrq+0x13c/0x1d8
[86365.660943] [<ffffffff80bd05d8>] write_sysrq_trigger+0x50/0x70
[86365.666780] [<ffffffff80a4af58>] proc_reg_write+0x70/0xa8
[86365.672184] [<ffffffff809e1c94>] __vfs_write+0x3c/0x108
[86365.677415] [<ffffffff809e2634>] vfs_write+0xa4/0x1a8
[86365.682471] [<ffffffff809e3148>] SyS_write+0x68/0x108
[86365.687528] [<ffffffff8087a190>] syscall_common+0x34/0x58
[86365.692930]
[86365.694420]
Code: 3c0381ba ac622b70 0000010f <03e00008> a0020000 00000000 67bdffe0 ffbf0018 ffb10010
[86365.704643] ---[ end trace 56d9f041ed6681dc ]---
[86365.774362] mtdoops: ready 78, 79 (no erase)
[86365.778657] Before the elf_core_copy_kernel_regs function call
[86365.784500] CPU: 0 PID: 817 Comm: bash Tainted: G D O 4.4.245-octeon-distro.git-v2.109-1-rc #1
[86365.794065] Stack : ffffffff81ba0000 0000000000000001 0000000000000000 0000000000000000
ffffffff81bf0000 ffffffff810d0000 0000000000000000 ffffffff81be91c0
0000000000000004 ffffffff810d0000 0000000000000000 0000000000000000
ffffffff81be91c0 ffffffff808e7b84 0000000000000006 ffffffff808e84bc
0000000000000000 0000000000000000 ffffffff81bb0000 ffffffff81ba2b78
ffffffff80ff5387 ffffffff80f359a8 80000003fa59de80 ffffffff81ba2b78
0000000000000331 0000000000000000 0000000000030002 80000003fa2adee8
80000003fd512420 80000003fc937430 80000003fc937548 ffffffff80b5b998
0000000000000000 ffffffff808e9634 ffffffffffffffc1 ffffffff80f359a8
0000000000000000 ffffffff80873280 0000000000000000 0000000000000000
...
[86365.859707] Call Trace:
[86365.862159] [<ffffffff80873280>] show_stack+0x98/0xb8
[86365.867216] [<ffffffff80b5b998>] dump_stack+0xe8/0x170
[86365.872364] [<ffffffff8091e468>] crash_save_cpu+0xa0/0x148
[86365.877854] [<ffffffff808845ec>] default_machine_crash_shutdown+0x3c/0xa0
[86365.884646] [<ffffffff8091e660>] crash_kexec+0x68/0xa8
[86365.889789] [<ffffffff808734e0>] die+0x138/0x158
[86365.894412] [<ffffffff808858a4>] __do_page_fault+0x584/0x598
[86365.900075] [<ffffffff8086c440>] ret_from_exception+0x0/0x18
[86365.905734]
[86365.907227] sizeof(*elfregs) = 400 and sizeof(*regs) = 400
[86365.912711] After the elf_core_copy_kernel_regs function call
[86365.918459] Sending IPI to other cpus...
[86375.921640] Will call new kernel at 0be3f6e0
[86375.925907] Bye ...
[86375.928019] Kernel unaligned instruction access[#2]:
[86375.932990] CPU: 0 PID: 817 Comm: bash Tainted: G D O 4.4.245-octeon-distro.git-v2.109-1-rc #1
[86375.942553] task: 80000003fa59de80 ti: 80000003fc934000 task.ti: 80000003fc934000
[86375.950032] $ 0 : 0000000000000000 000000000be3f7b4 ffffffff0be3f7b4 0000000000000040
[86375.958085] $ 4 : 0000000000000000 0000000000000000 0000000000000001 800000040f9d1248
[86375.966137] $ 8 : 0000000000003ffe 0000000000000257 ffffffff81bf0000 ffffffff810d0000
[86375.974188] $12 : ffffffff900fe000 ffffffff80000000 ffffffff80884250 0000000000000000
[86375.982239] $16 : 80000003f981c008 800000000be3f6e0 0000000000000004 0000000000000004
[86375.990289] $20 : 80000003fa2ade80 0000000000000001 0000000000030002 80000003fa2adee8
[86375.998340] $24 : ffffffff81bf0000 ffffffff80bddc68
[86376.006392] $28 : 80000003fc934000 80000003fc937840 80000003fd512420 800000000cd780e0
[86376.014443] Hi : 0000000000e20d34
[86376.018015] Lo : 5a1cac0832335927
[86376.021589] epc : ffffffff0be3f7b4 0xffffffff0be3f7b4
[86376.026811] ra : 800000000cd780e0 0x800000000cd780e0
[86376.032032] Status: 10008ce2 KX SX UX KERNEL EXL
[86376.036796] Cause : 40808c10 (ExcCode 04)
[86376.040801] BadVA : ffffffff0be3f7b4
[86376.044373] PrId : 000d9101 (Cavium Octeon II)
[86376.048899] Modules linked in: ecc_driver_edac(O) ecc_driver_core(O) netconsole octeon_edac_pci octeon_edac_l2c octeon_edac_pc octeon_edac_lmc edac_core mtdoops isofs ehci_platform ehci_hcd physmap autofs4 nfsv3 nfs_acl ipmi_watchdog 8021q garp stp mrp llc ipmi_si ipmi_devintf ipmi_msghandler nfs lockd grace sunrpc octeon_ethernet ethernet_mem igb hwmon i2c_algo_bit e1000e ptp pps_core
[86376.083451] Process bash (pid: 817, threadinfo=80000003fc934000, task=80000003fa59de80, tls=000000ffea983f60)
[86376.093360] Stack : ffffffff810d0000 80000003fc937b20 ffffffff81bf0000 ffffffff8091e668
0000000000000000 ffffffff80bd04b4 0000000000000001 ffffffff81ba0000
0000000000000063 80000000a5130558 80000000a5132b68 0000000000000005
ffffffffc027e260 0000000000000221 ffffffff81bf0000 ffffffff810d0000
0000000000000000 0000000000000000 0000000000000220 0000000000000000
ffffffff8111ee80 0000000000000063 0000000000000004 0000000000000000
ffffffff810d0000 ffffffff8111eb60 ffffffff81120000 000000ffffb69cd4
0000000000000006 ffffffff80bddc68 ffffffff81bb0000 ffffffff809f2328
80000003fc934000 80000003fc937cb0 000000ffea89e6c8 ffffffff80bd04b4
0000000010008ce3 000000000061cb2f df3b645a1d20297a 0000000000000000
...
[86376.158859] Call Trace:
[86376.161320] [<ffffffff8091e668>] crash_kexec+0x70/0xa8
[86376.166458] [<ffffffff80bd04b4>] __handle_sysrq+0x13c/0x1d8
[86376.172032] [<ffffffffc027e260>] write_msg+0x0/0x198 [netconsole]
[86376.178127] [<ffffffff80bddc68>] dw8250_serial_outq+0x0/0x118
[86376.183873] [<ffffffff809f2328>] path_openat+0x1e8/0x1120
[86376.189270] [<ffffffff80bd04b4>] __handle_sysrq+0x13c/0x1d8
[86376.194841] [<ffffffff80bcfe5c>] sysrq_handle_crash+0x1c/0x28
[86376.200585] [<ffffffff80bd04a4>] __handle_sysrq+0x12c/0x1d8
[86376.206159] [<ffffffff808734e0>] die+0x138/0x158
[86376.210776] [<ffffffff808858a4>] __do_page_fault+0x584/0x598
[86376.216437] [<ffffffff80bd78c0>] serial8250_console_putchar+0x0/0x50
[86376.222790] [<ffffffff80bda388>] serial8250_console_write+0x118/0x2b0
[86376.229230] [<ffffffff8086c440>] ret_from_exception+0x0/0x18
[86376.234887] [<ffffffff80bd04b4>] __handle_sysrq+0x13c/0x1d8
[86376.240459] [<ffffffffc027e260>] write_msg+0x0/0x198 [netconsole]
[86376.246551] [<ffffffff80bddc68>] dw8250_serial_outq+0x0/0x118
[86376.252297] [<ffffffff809f2328>] path_openat+0x1e8/0x1120
[86376.257694] [<ffffffff80bd04b4>] __handle_sysrq+0x13c/0x1d8
[86376.263264] [<ffffffff80bcfe5c>] sysrq_handle_crash+0x1c/0x28
[86376.269008] [<ffffffff80bd04a4>] __handle_sysrq+0x12c/0x1d8
[86376.274579] [<ffffffff80bd05d8>] write_sysrq_trigger+0x50/0x70
[86376.280411] [<ffffffff80a4af58>] proc_reg_write+0x70/0xa8
[86376.285811] [<ffffffff809e1c94>] __vfs_write+0x3c/0x108
[86376.291035] [<ffffffff809e2634>] vfs_write+0xa4/0x1a8
[86376.296085] [<ffffffff809e838c>] SyS_newfstat+0x3c/0x50
[86376.301309] [<ffffffff809e3148>] SyS_write+0x68/0x108
[86376.306360] [<ffffffff8087a190>] syscall_common+0x34/0x58
[86376.311757] [<ffffffff809e8020>] cp_new_stat+0xb0/0x188
[86376.316981] [<ffffffff80b6dde8>] __bzero+0x84/0x13c
[86376.321857] [<ffffffff809e8350>] SyS_newfstat+0x0/0x50
[86376.326993]
[86376.328481]
Code:
telnet>
Connection closed.
[root@CFPU-0(BCNBlr104_109) /root]
# telnet lmp-1-1-1 3004
