Hi Sarah,
many thanks for your answer, not sure about that,
but it seems that you have aimed the spotlight in the correct direction:
setting CONFIG_PCI_DEBUG=y I found out that the crash
is due to an ARM processor fault,
namely, a data abort due to a PCI access failure:
this happens when, in the very first lines of xhci_event_ring_work(),
your code references the status register
within the XHCI-HCD operational registers set
therefore, we definitely have no wandering pointers:
most likely, the PCIe transaction for accessing that register
went bad, for some reason, either hardware- or driver-related:
the ti816x_pcie.c driver module is maintained directly from TI,
so I have to refer to TI folks directly...
...unless, of course, you have some more hints for me
about debugging the PCIe side of an USB3 hub :-)
best regards,
Francesco
Il 14/06/2011 02:04, Sarah Sharp ha scritto:
On Fri, Jun 10, 2011 at 02:08:27PM +0200, Francesco Castelli wrote:
Hi Sarah,
let me explain better this time:
-first of all, I work as an Embedded Linux consultant
on a prototype board with a Texas Instruments TI8168 device
(brief data sheet here: http://focus.ti.com/lit/ds/sprs614/sprs614.pdf)
-the PCIe controller embedded into TI8168 is connected
to the (TI again) TUSB7320 2-ports xHCI-compliant USB3 controller
(data manual here: http://focus.ti.com/lit/ds/symlink/tusb7320.pdf)
-TI delivers a complete TI81XX SDK covering all the (many) onboard cores:
the ARM Cortex-A8 is supported via patching Linux Kernel 2.6.37
basicly with the TI8168 chip support and the TI EVM board support,
PLEASE NOTE that the drivers/usb subtree has NOT been patched in any way
Ok, good to know.
-I ran 'make menuconfig' after selecting my boards' defconfig,
adding the following lines:
CONFIG_USB_DEBUG=y
CONFIG_USB_XHCI_HCD=m
CONFIG_USB_XHCI_HCD_DEBUGGING=y
-my runtime routine is to prevent autoloading xhci-hcd.ko via blacklisting
and load it later via modprobe, I don't know if this helps, anyway
now, among the various issues I raised in the previous email,
I would like to focus on what I think is the major problem: the random crash
that happens with NO (!!!) USB peripheral plugged in
(BTW 'gadget' is the misleading term I used in place of 'peripheral')
please find attached a single TXT file with three sections of dmesg dumps:
-the first one refers to when I load xhci-hcd.ko via insmod
Ok, that looks normal since you enabled CONFIG_USB_XHCI_HCD_DEBUGGING.
-the second one appears randomly and looks like an activity report
from the hub;
Yes, that's the polling loop that runs every minute. I probably should
change it to check the enqueue and dequeue pointers from the previous
run, and not print anything if nothing changed, but it's normal.
-the last one is the crash itself, an illegal access at location 0x00001008
which is not addressable by the ARM processor
Yeah, that's probably a NULL pointer dereference.
####################################################################
# fault that happened without NO USB peripheral plugged in - BEGIN #
####################################################################
xhci_hcd 0000:01:00.0: Poll event ring: 14496
Unhandled fault: Precise External Abort on non-linefetch (0x1008) at 0x8f140024
Internal error: : 1008 [#1]
last sysfs file: /sys/devices/pci0000:00/0000:00:00.0/0000:01:00.0/usb3/idVendor
Modules linked in: aett ti8168_vpssm3 ti8168_videom3 ti8168_c674xplus TI81xx_hdmi syslink xhci_hcd ipv6 aeremote rvphoenix_board xNCI hal_xNCI pSOSk gmem
CPU: 0 Not tainted (2.6.37 #1)
PC is at xhci_event_ring_work+0x34/0x240 [xhci_hcd]
LR is at release_console_sem+0x198/0x1ac
pc : [<7f0825cc>] lr : [<8006516c>] psr: 20000193
sp : 80477e70 ip : 80477d58 fp : 80477e94
r10: 804d934c r9 : 804d914c r8 : 20000113
r7 : 00000100 r6 : 804d8540 r5 : 8ac6c0dc r4 : 000000a0
r3 : 8f140020 r2 : 00000001 r1 : 0000fa58 r0 : 00000034
Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c5387d Table: 8acbc019 DAC: 00000017
Process swapper (pid: 0, stack limit = 0x804762e8)
Stack: (0x80477e70 to 0x80478000)
7e60: 000000a0 8ac6c564 804d8540 00000100
7e80: 80476000 804d914c 80477ee4 80477e98 80071260 7f0825a4 8008a518 8008a358
7ea0: 8ac6c0dc 7f082598 804d8f4c 804d8d4c 80477eb0 80477eb0 00989680 00000043
7ec0: 80476000 00000001 00000004 00000001 0000000a 00000100 80477f24 80477ee8
7ee0: 8006a42c 80071088 80477f1c 80477ef8 80477f0c 804d8304 8004f574 00000043
7f00: 00000000 8002d8ac 8047a0f4 8002bfa8 413fc082 0000001f 80477f34 80477f28
7f20: 8006a55c 8006a354 80477f4c 80477f38 80037080 8006a520 ffffffff fa200000
7f40: 80477fa4 80477f50 80361d74 8003700c 81600181 40000013 80477f98 00000814
7f60: 80476000 804b7380 8002d8ac 8047a0f4 8002bfa8 413fc082 0000001f 80477fa4
7f80: 80477f98 80477f98 80042f5c 80042f60 80000013 ffffffff 80477fbc 80477fa8
7fa0: 80043500 80042f30 804ff83c 804b7380 80477fcc 80477fc0 80357474 800434bc
7fc0: 80477ff4 80477fd0 80008c64 80357420 800087c4 00000000 00000000 8002d8b0
7fe0: 10c53c7d 804b749c 00000000 80477ff8 80008034 80008a20 00000000 00000000
Backtrace:
[<7f082598>] (xhci_event_ring_work+0x0/0x240 [xhci_hcd]) from [<80071260>] (run_timer_softirq+0x1e4/0x2bc)
r9:804d914c r8:80476000 r7:00000100 r6:804d8540 r5:8ac6c564
r4:000000a0
[<8007107c>] (run_timer_softirq+0x0/0x2bc) from [<8006a42c>] (__do_softirq+0xe4/0x1cc)
[<8006a348>] (__do_softirq+0x0/0x1cc) from [<8006a55c>] (irq_exit+0x48/0x94)
[<8006a514>] (irq_exit+0x0/0x94) from [<80037080>] (asm_do_IRQ+0x80/0xa0)
[<80037000>] (asm_do_IRQ+0x0/0xa0) from [<80361d74>] (__irq_svc+0x34/0xa0)
Exception stack(0x80477f50 to 0x80477f98)
7f40: 81600181 40000013 80477f98 00000814
7f60: 80476000 804b7380 8002d8ac 8047a0f4 8002bfa8 413fc082 0000001f 80477fa4
7f80: 80477f98 80477f98 80042f5c 80042f60 80000013 ffffffff
r5:fa200000 r4:ffffffff
[<80042f24>] (default_idle+0x0/0x40) from [<80043500>] (cpu_idle+0x50/0x90)
[<800434b0>] (cpu_idle+0x0/0x90) from [<80357474>] (rest_init+0x60/0x78)
r5:804b7380 r4:804ff83c
[<80357414>] (rest_init+0x0/0x78) from [<80008c64>] (start_kernel+0x250/0x2a4)
[<80008a14>] (start_kernel+0x0/0x2a4) from [<80008034>] (stext+0x34/0x3c)
r5:804b749c r4:10c53c7d
Code: eb45c29a e10f8000 f10c0080 e5953004 (e5934004)
---[ end trace b32101bf3e82059a ]---
Kernel panic - not syncing: Fatal exception in interrupt
Backtrace:
[<80045ce4>] (dump_backtrace+0x0/0x10c) from [<8035fbc0>] (dump_stack+0x18/0x1c)
r7:7f0825d0 r6:80477d1f r5:7f0825ce r4:804b79e0
[<8035fba8>] (dump_stack+0x0/0x1c) from [<8035fc24>] (panic+0x60/0x17c)
[<8035fbc4>] (panic+0x0/0x17c) from [<80046074>] (die+0x284/0x2d8)
r3:00000100 r2:8041ea52 r1:00000000 r0:8040dbc6
[<80045df0>] (die+0x0/0x2d8) from [<80046188>] (arm_notify_die+0x5c/0x60)
[<8004612c>] (arm_notify_die+0x0/0x60) from [<800372f0>] (do_DataAbort+0x88/0x9c)
r5:8047a7c0 r4:00000007
[<80037268>] (do_DataAbort+0x0/0x9c) from [<80361d2c>] (__dabt_svc+0x4c/0x60)
Exception stack(0x80477e28 to 0x80477e70)
7e20: 00000034 0000fa58 00000001 8f140020 000000a0 8ac6c0dc
7e40: 804d8540 00000100 20000113 804d914c 804d934c 80477e94 80477d58 80477e70
7e60: 8006516c 7f0825cc 20000193 ffffffff
r8:20000113 r7:00000100 r6:804d8540 r5:80477e5c r4:ffffffff
[<7f082598>] (xhci_event_ring_work+0x0/0x240 [xhci_hcd]) from [<80071260>] (run_timer_softirq+0x1e4/0x2bc)
r9:804d914c r8:80476000 r7:00000100 r6:804d8540 r5:8ac6c564
r4:000000a0
[<8007107c>] (run_timer_softirq+0x0/0x2bc) from [<8006a42c>] (__do_softirq+0xe4/0x1cc)
[<8006a348>] (__do_softirq+0x0/0x1cc) from [<8006a55c>] (irq_exit+0x48/0x94)
[<8006a514>] (irq_exit+0x0/0x94) from [<80037080>] (asm_do_IRQ+0x80/0xa0)
[<80037000>] (asm_do_IRQ+0x0/0xa0) from [<80361d74>] (__irq_svc+0x34/0xa0)
Exception stack(0x80477f50 to 0x80477f98)
7f40: 81600181 40000013 80477f98 00000814
7f60: 80476000 804b7380 8002d8ac 8047a0f4 8002bfa8 413fc082 0000001f 80477fa4
7f80: 80477f98 80477f98 80042f5c 80042f60 80000013 ffffffff
r5:fa200000 r4:ffffffff
[<80042f24>] (default_idle+0x0/0x40) from [<80043500>] (cpu_idle+0x50/0x90)
[<800434b0>] (cpu_idle+0x0/0x90) from [<80357474>] (rest_init+0x60/0x78)
r5:804b7380 r4:804ff83c
[<80357414>] (rest_init+0x0/0x78) from [<80008c64>] (start_kernel+0x250/0x2a4)
[<80008a14>] (start_kernel+0x0/0x2a4) from [<80008034>] (stext+0x34/0x3c)
r5:804b749c r4:10c53c7d
####################################################################
# fault that happened without NO USB peripheral plugged in - END #
####################################################################
That is very strange looking. Have you tried to run the oops through
scripts/markup-oops.pl to see where exactly the xHCI driver is oopsing?
If nothing changed in the driver state, why would this function
dereference a NULL pointer on some random iteration through
the polling loop in xhci_event_ring_work? I'm baffled.
Did the xHCI host controller disconnect from the PCIe bus some time
before, or was there any abnormal activity before this crash?
Sarah Sharp
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
