Hi,
We've got a custom AM3359 board and we occasionally find that some devices start behaving in an unexpected manner, causing some of our
processes to stop or even crash.
Analysing these devices (running Kernels 4.1.13 / 4.10.17) showed us that the gp_timer ( based on DMTIMER2, clock_event_device, default setting)
has stopped running. We do not know what causes the timer to stop, but the affected devices display the following symptoms:
- DMTIMER2 Registers:
- The counter register TCRR is neither 0, nor has it got the values or the Timer Load Register(TLDR) or Timer Trigger Register (TTGR) .
- The IRQStatus register indicates that there are no pending timer interrupts
- The ST bit of the TCLR register is 0 (Instead of 1)
- Obviously the jiffies count has also stopped.
- On Kernel 4.1.13 additionally observed behaviour:
- the devices is not responding for 115 seconds ("closed period"),
followed by 65s of "normal operation" ("open period") iteratively.
- the system time is always reset at the start of the "open period" to a
time prior to when the problem occurred.
Setting the ST bit manually to 0 stops the gp_timer and displays all the above mentioned symptoms, but we do not know why or who sets it
to 0 instead of 1 in the error case.
We can imagine the following use cases that could trigger it:
- Another process stops the timer accidentally by clearing the ST bit of the TCLR register.
- Setting the ST bit of the TCLR register after loading the TCRR register goes wrong ( in omap2_gp_timer_set_next_event() ) e.g. interrupt, posted mode.
We already tried the following methods to trigger the error, but no success so far:
* Software Tests:
- Overload the system with posix timer threads
* Hardware -Tests:
- EMC Tests on the device
- Random bursts on the RxD signal of UART0 interface.
- NMI interrupts (EXTINTn signal)
- Feed noise into the 32kHz oscillator circuit (https://e2e.ti.com/support/arm/sitara_arm/f/791/t/270632)
- Feed noise into the 24KHz timer oscillator circuit (idem)
- Increasing the DMA measurement interrupt rate up to 6 kHz
- Toggling the JTAG pins EMU0 and EMU1
Also the following paths have been investigated:
- advisory 1.0.39 in http://www.ti.com/lit/er/sprz360i/sprz360i.pdf, but there exists already a workaround in the Linux kernel.
Has anybody already seen something similar, or got an idea what else we should investigate to trace this problem?
Here some more details to our environment:
- Custom board with AM3359 and FPGA (with GPMC/DMA)
- Kernel is based on linux-yocto jethro(4.1.13)/pyro(4.10.17)
- We use the standard device tree for the clock configuration (am33xx.dtsi)
Thanks in advance
Best regards
Isak Lichtenstein
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
