Hi, I found a problem on crash-4.0-2.12.
Summary:
bt command does not show stack traces of some CPUs.
Condition:
This problem happens only on ia64 machine.
There are two conditions to reproduce this problem.
1) Diskdump is executed via OS_INIT.
2) The machine has more than 8 CPUs.
Details:
When I executed bt command for vmcore which was created
on the 32 CPU machine, bt didn't show stack traces of some CPU.
Please see attached file(bt_failed.txt). Stack traces from CPU0 to
CPU7 are showed normally, but stack traces from CPU8 to CPU31 are not.
(Please don't worry about a message "unwind: bsp (xxxxxxxxx) out of
range". This is a problem of our platform.)
Cause:
I found a bug in ia64.c.
2679 ms->ia64_init_stack_size = get_array_length("ia64_init_stack",
2680 NULL, 0);
get_array_length() gets the length of stack of OS_INIT, and the
length is stored at ms->ia64_init_stack_size. However, the value
which get_array_length gives is different from actual stack length
because "ia64_init_stack" is declared like this:
u64 ia64_init_stack[NR_CPUS*KERNEL_STACK_SIZE/8];
Therefore, correct length of a stack is this:
get_array_length("ia64_init_stack", NULL, 0) * sizeof(u64)
I don't know how to fix, but it seems that attached patch
(ia64.c.patch) corrects this problem.
Another attached patch(test.patch) also seems to fix the problem,
but I don't know which is better.
Regards,
Takao Indoh
KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL.guest/vmlinux
DUMPFILE: ./vmcore
CPUS: 32
DATE: Sat Nov 12 12:56:07 2005
UPTIME: 00:04:25
LOAD AVERAGE: 1.84, 1.51, 0.66
TASKS: 294
NODENAME: build.fujitsu.com
RELEASE: 2.6.9-22.EL.guest
VERSION: #1 SMP Tue Nov 8 17:16:52 JST 2005
MACHINE: ia64 (1600 Mhz)
MEMORY: 1023.5 GB
PANIC: ""
PID: 0
COMMAND: "swapper"
TASK: e00001c004080000 (1 of 32) [THREAD_INFO: e00001c004080dc0]
CPU: 5
STATE: TASK_RUNNING (INIT)
crash> bt -a
PID: 0 TASK: a0000001006c4000 CPU: 0 COMMAND: "swapper"
#0 [BSP:a0000001006c5018] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047c7530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120010 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e0000140209c0330 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a0000001000e54e0 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: a0000001006cbe20 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: a0000001006cbd80 R13: a0000001006c4000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126010
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 000000000013bc00 R25: 000000007faabd00
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffe9b74ed80000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 1000be7bcccccc837a100 F11: 1003e0000000000000000
#1 [BSP:a0000001006c4fc0] v at e00000007fd20300
#2 [BSP:a0000001006c4f70] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xa0000001006c4b80) out of range [0xa0000001006c4dc0-0xa0000001006c5018]
PID: 0 TASK: e000024004100000 CPU: 1 COMMAND: "swapper"
#0 [BSP:e000024004100f30] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047cf530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120010 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e000020020270018 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a000000100015c40 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: e000024004107e30 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: e000024004107d90 R13: e000024004100000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126010
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 00000000003dbc00 R25: 80000000ffed93d0
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffe6cfd6e00000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 10009835ffffffd66ea00 F11: 1003e0000000000000000
#1 [BSP:e000024004100ee0] v at e00000007fd20300
#2 [BSP:e000024004100e90] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xe000024004100aa0) out of range [0xe000024004100dc0-0xe000024004100f30]
PID: 0 TASK: e000014004070000 CPU: 2 COMMAND: "swapper"
#0 [BSP:e000014004070f30] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047d7530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120010 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e000020020618018 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a0000001000e54e0 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: e000014004077e30 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: e000014004077d90 R13: e000014004070000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126010
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 00000000003f7c00 R25: 80000000ffed93d0
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffe9c03e760000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 1000bf308ccccc7fe7040 F11: 1003e0000000000000000
#1 [BSP:e000014004070ee0] v at e00000007fd20300
#2 [BSP:e000014004070e90] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xe000014004070aa0) out of range [0xe000014004070dc0-0xe000014004070f30]
PID: 0 TASK: e000024004108000 CPU: 3 COMMAND: "swapper"
#0 [BSP:e000024004108f30] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047df530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120030 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e000024020638018 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a0000001000e54e0 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: e00002400410fe30 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: e00002400410fd90 R13: e000024004108000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126030
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 00000000003fe400 R25: 80000000ffed93d0
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffeac196b60000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 1000cf4bbfffffb290840 F11: 1003e0000000000000000
#1 [BSP:e000024004108ee0] v at e00000007fd20300
#2 [BSP:e000024004108e90] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xe000024004108aa0) out of range [0xe000024004108dc0-0xe000024004108f30]
PID: 0 TASK: e00000017c8b0000 CPU: 4 COMMAND: "swapper"
#0 [BSP:e00000017c8b0f30] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047e7530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120030 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e000028020608018 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a0000001000e54e0 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: e00000017c8b7e30 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: e00000017c8b7d90 R13: e00000017c8b0000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126030
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 00000000003bfc00 R25: 80000000ffed93d0
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffe9aafb020000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 1000bd82733332eececc0 F11: 1003e0000000000000000
#1 [BSP:e00000017c8b0ee0] v at e00000007fd20300
#2 [BSP:e00000017c8b0e90] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xe00000017c8b0aa0) out of range [0xe00000017c8b0dc0-0xe00000017c8b0f30]
PID: 0 TASK: e00001c004080000 CPU: 5 COMMAND: "swapper"
#0 [BSP:e00001c004081120] start_disk_dump at a000000200699ca0
#1 [BSP:e00001c004081100] try_crashdump at a0000001000ab710
#2 [BSP:e00001c0040810d0] start_dump_osinit at a00000010004cae0
#3 [BSP:e00001c0040810a8] unw_init_running at a00000010000faf0
#4 [BSP:e00001c004080f30] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047ef530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120010 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e00000017c9a0018 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a0000001000e54e0 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: e00001c004087e30 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: e00001c004087d90 R13: e00001c004080000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126010
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 00000000003c2c00 R25: 80000000ffed93d0
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffe9c5f0760000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 1000bfa3bfffffb0d3040 F11: 1003e0000000000000000
#5 [BSP:e00001c004080ee0] v at e00000007fd20300
#6 [BSP:e00001c004080e90] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xe00001c004080aa0) out of range [0xe00001c004080dc0-0xe00001c004080f30]
PID: 0 TASK: e00002c004158000 CPU: 6 COMMAND: "swapper"
#0 [BSP:e00002c004158f30] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047f7530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120030 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e000014020628018 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a0000001000e54e0 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: e00002c00415fe30 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: e00002c00415fd90 R13: e00002c004158000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126030
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 000000000029bc00 R25: 80000000ffed93d0
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffe9b8399e0000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 1000be8e5999994fe8f40 F11: 1003e0000000000000000
#1 [BSP:e00002c004158ee0] v at e00000007fd20300
#2 [BSP:e00002c004158e90] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xe00002c004158aa0) out of range [0xe00002c004158dc0-0xe00002c004158f30]
PID: 0 TASK: e000018004088000 CPU: 7 COMMAND: "swapper"
#0 [BSP:e000018004088f30] ia64_init_handler at a00000010004f4c0
EFRAME: e0000000047ff530
B0: a000000100015f60 CR_IIP: e00000007fd20300
CR_IPSR: 0000101008120010 CR_IFS: 000000000000050a
AR_PFS: 000000000000050f AR_RSC: 0000000000000000
AR_UNAT: 0000000000000000 AR_RNAT: 0000000000000000
AR_CCV: e000018020618018 AR_FPSR: 0009804c8a70433f
LOADRS: 0000000000000000 AR_BSPSTORE: 0000000000000000
B6: a0000001000e54e0 B7: e00000007fd08010
PR: 80000000ff6d1125 R1: a00000010099d150
R2: e00001800408fe30 R3: 000000000000050f
R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000
R12: e00001800408fd90 R13: e000018004088000
R14: e00000007fd20300 R15: 800000000000050a
R16: a000000100015f80 R17: 0000101008126010
R18: 0000000000000004 R19: 80000000ff6d1125
R20: 88c0001c00000000 R21: 0000000000000007
R22: 0000000000000000 R23: 0000000000000609
R24: 00000000003ffc00 R25: 80000000ffed93d0
R26: 0000000000000000 R27: 0000000000000000
R28: 0000000000000000 R29: 0000000000000000
R30: 0000000000000000 R31: 0000000000000000
F6: 0fffbccccccccc8c00000 F7: 0ffead19c9a0000000000
F8: 000000000000000000000 F9: 100068000000000000000
F10: 1000d847eccccca2e0ae0 F11: 1003e0000000000000000
#1 [BSP:e000018004088ee0] v at e00000007fd20300
#2 [BSP:e000018004088e90] ia64_pal_call_static at a000000100015f60
bt: unwind: bsp (0xe000018004088aa0) out of range [0xe000018004088dc0-0xe000018004088f30]
PID: 0 TASK: e000024004118000 CPU: 8 COMMAND: "swapper"
PID: 0 TASK: e000014020528000 CPU: 9 COMMAND: "swapper"
PID: 0 TASK: e000020004108000 CPU: 10 COMMAND: "swapper"
PID: 0 TASK: e00002c004108000 CPU: 11 COMMAND: "swapper"
PID: 0 TASK: e00001c0202d8000 CPU: 12 COMMAND: "swapper"
PID: 0 TASK: e000028004108000 CPU: 13 COMMAND: "swapper"
PID: 0 TASK: e000014020538000 CPU: 14 COMMAND: "swapper"
PID: 0 TASK: e000024004130000 CPU: 15 COMMAND: "swapper"
PID: 0 TASK: e00000017c8d0000 CPU: 16 COMMAND: "swapper"
PID: 0 TASK: e00001c020548000 CPU: 17 COMMAND: "swapper"
PID: 0 TASK: e00002c004120000 CPU: 18 COMMAND: "swapper"
PID: 0 TASK: e00001c020550000 CPU: 19 COMMAND: "swapper"
PID: 0 TASK: e000028004118000 CPU: 20 COMMAND: "swapper"
PID: 0 TASK: e00001c020558000 CPU: 21 COMMAND: "swapper"
PID: 0 TASK: e000028004120000 CPU: 22 COMMAND: "swapper"
PID: 0 TASK: e000014020558000 CPU: 23 COMMAND: "swapper"
PID: 0 TASK: e000024020570000 CPU: 24 COMMAND: "swapper"
PID: 0 TASK: e00000017c8f0000 CPU: 25 COMMAND: "swapper"
PID: 0 TASK: e00001c020568000 CPU: 26 COMMAND: "swapper"
PID: 0 TASK: e00002c0040c0000 CPU: 27 COMMAND: "swapper"
PID: 0 TASK: e000018020560000 CPU: 28 COMMAND: "swapper"
PID: 0 TASK: e000024020580000 CPU: 29 COMMAND: "swapper"
PID: 0 TASK: e000014020570000 CPU: 30 COMMAND: "swapper"
PID: 0 TASK: e000020020560000 CPU: 31 COMMAND: "swapper"
Attachment:
ia64.c.patch
Description: Binary data
Attachment:
test.patch
Description: Binary data
