Hi,
Below are the RCU logs, I have tried enabling RCU debugging but didn't recieve much help.
root@ls1021aqds:~#
root@ls1021aqds:~# INFO: rcu_preempt self-detected stall on CPU { 0} (t=2100 jiffies g=301 c=300 q=1)
CPU: 0 PID: 1099 Comm: irq/177-eth0_g0 Not tainted 3.12.19-rt30+ #2
[<800109b1>] (unwind_backtrace+0x1/0x88) from [<8000e697>] (show_stack+0xb/0xc)
[<8000e697>] (show_stack+0xb/0xc) from [<803023a3>] (dump_stack+0x4b/0x84)
[<803023a3>] (dump_stack+0x4b/0x84) from [<80051c8d>] (rcu_check_callbacks+0x145/0x468)
[<80051c8d>] (rcu_check_callbacks+0x145/0x468) from [<800202b7>] (update_process_times+0x2f/0x48)
[<800202b7>] (update_process_times+0x2f/0x48) from [<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38)
[<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38) from [<800482d5>] (tick_sched_timer+0x2d/0x50)
[<800482d5>] (tick_sched_timer+0x2d/0x50) from [<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4)
[<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4) from [<8002d319>] (hrtimer_interrupt+0x101/0x268)
[<8002d319>] (hrtimer_interrupt+0x101/0x268) from [<80257fef>] (arch_timer_handler_phys+0x1f/0x22)
[<80257fef>] (arch_timer_handler_phys+0x1f/0x22) from [<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50)
[<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50) from [<8003d2bf>] (generic_handle_irq+0x13/0x1c)
[<8003d2bf>] (generic_handle_irq+0x13/0x1c) from [<8000cb7d>] (handle_IRQ+0x3d/0x60)
[<8000cb7d>] (handle_IRQ+0x3d/0x60) from [<800083df>] (gic_handle_irq+0x2b/0x44)
[<800083df>] (gic_handle_irq+0x2b/0x44) from [<80304c9b>] (__irq_svc+0x3b/0x94)
Exception stack(0xba9b7c58 to 0xba9b7ca0)
7c40: 804d6400 ba9b7ce8
7c60: 00000000 00000004 ba84be80 ba84be80 804d7268 8050a3c0 be9f2000 00000000
7c80: bf8c9900 be3b6e60 00891000 ba9b7ca0 802a2e61 80303d5a 00030133 ffffffff
[<80304c9b>] (__irq_svc+0x3b/0x94) from [<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0)
[<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0) from [<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c)
[<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c) from [<80294f73>] (dst_destroy+0x3b/0xa4)
[<80294f73>] (dst_destroy+0x3b/0xa4) from [<80295395>] (dst_release+0x41/0x50)
[<80295395>] (dst_release+0x41/0x50) from [<80287965>] (skb_release_head_state+0x15/0x70)
[<80287965>] (skb_release_head_state+0x15/0x70) from [<802897f7>] (skb_release_all+0x9/0x1a)
[<802897f7>] (skb_release_all+0x9/0x1a) from [<80289811>] (__kfree_skb+0x9/0x88)
[<80289811>] (__kfree_skb+0x9/0x88) from [<8029e521>] (pfifo_fast_enqueue+0x65/0x78)
[<8029e521>] (pfifo_fast_enqueue+0x65/0x78) from [<8029179f>] (dev_queue_xmit+0x1c3/0x310)
[<8029179f>] (dev_queue_xmit+0x1c3/0x310) from [<802a86eb>] (ip_finish_output+0x2a7/0x304)
[<802a86eb>] (ip_finish_output+0x2a7/0x304) from [<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338)
[<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338) from [<8028f94d>] (netif_receive_skb+0x3d/0x74)
[<8028f94d>] (netif_receive_skb+0x3d/0x74) from [<8028fd87>] (napi_gro_receive+0x2b/0x60)
[<8028fd87>] (napi_gro_receive+0x2b/0x60) from [<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4)
[<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4) from [<801f2483>] (gfar_poll_rx_sq+0x17/0x68)
[<801f2483>] (gfar_poll_rx_sq+0x17/0x68) from [<8029071b>] (net_rx_action+0x87/0x120)
[<8029071b>] (net_rx_action+0x87/0x120) from [<8001b7f5>] (do_current_softirqs+0x105/0x1cc)
[<8001b7f5>] (do_current_softirqs+0x105/0x1cc) from [<8001b8f9>] (local_bh_enable+0x3d/0x60)
[<8001b8f9>] (local_bh_enable+0x3d/0x60) from [<8003df2f>] (irq_forced_thread_fn+0x2d/0x30)
[<8003df2f>] (irq_forced_thread_fn+0x2d/0x30) from [<8003e09f>] (irq_thread+0x8f/0xf8)
[<8003e09f>] (irq_thread+0x8f/0xf8) from [<8002a9ad>] (kthread+0x5d/0x6c)
[<8002a9ad>] (kthread+0x5d/0x6c) from [<8000c35d>] (ret_from_fork+0x11/0x34)
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=302, c=301, q=1)
INFO: Stall ended before state dump start
Thanks
On Wed, Jan 14, 2015 at 9:50 AM, Vignesh Radhakrishnan <vignesh1192@xxxxxxxxx> wrote:
Thanks and regards,2. Enable RCU_DEBUGGING options from config and get data points from those to see what is happening using ftrace.This seems to have hit RCU stalls here which got solved while detecting this. This happens when a grace period is not completed for more than 21 seconds (modifiable timeout from config).1. Enable lockdep and use debug_show_all_locks() API inside the stall detection code to see who are holding the read locks and figure out why they are not releasing it
If you are able to reproduce this, can you try the following things :
Vignesh RadhakrishnanOn Wed, Jan 14, 2015 at 9:00 AM, meenakshi aggarwal <meenu23688@xxxxxxxxx> wrote:_______________________________________________Hi,
I am receiving following kernel warning while running kernel
“3.12.19-rt30+ #6 SMP PREEMPT RT Tue Jan 13 15:35:39 IST 2015 armv7l GNU/Linux” on ARM v7 platform.
Kindly help.
[ 1044.465822] libphy: mdio@2d24000:02 - Link is Up - 1000/Full
[ 1058.805821] libphy: mdio@2d24000:00 - Link is Up - 1000/Full
[ 1662.764689] ------------[ cut here ]------------
[ 1662.768867] WARNING: CPU: 1 PID: 18 at net/sched/sch_generic.c:279 dev_watchdog+0x137/0x198()
[ 1662.777581] NETDEV WATCHDOG: eth1 (fsl-gianfar): transmit queue 0 timed out
[ 1662.876129] Modules linked in:
[ 1662.885557] CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 3.12.19-rt30+ #1
[ 1662.885609] [<80010991>] (unwind_backtrace+0x1/0x88) from [<8000e677>] (show_stack+0xb/0xc)
[ 1662.885628] [<8000e677>] (show_stack+0xb/0xc) from [<803024d3>] (dump_stack+0x4b/0x84)
[ 1662.885646] [<803024d3>] (dump_stack+0x4b/0x84) from [<80018d4d>] (warn_slowpath_common+0x45/0x64)
[ 1662.885660] [<80018d4d>] (warn_slowpath_common+0x45/0x64) from [<80018d87>] (warn_slowpath_fmt+0x1b/0x24)
[ 1662.885678] [<80018d87>] (warn_slowpath_fmt+0x1b/0x24) from [<8029e5b7>] (dev_watchdog+0x137/0x198)
[ 1662.885697] [<8029e5b7>] (dev_watchdog+0x137/0x198) from [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54)
[ 1662.885712] [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54) from [<8001ff97>] (run_timer_softirq+0x107/0x140)
[ 1662.885725] [<8001ff97>] (run_timer_softirq+0x107/0x140) from [<8001b7ad>] (do_current_softirqs+0x105/0x1cc)
[ 1662.885737] [<8001b7ad>] (do_current_softirqs+0x105/0x1cc) from [<8001b9cb>] (run_ksoftirqd+0x23/0x38)
[ 1662.885754] [<8001b9cb>] (run_ksoftirqd+0x23/0x38) from [<8002fb15>] (smpboot_thread_fn+0x175/0x188)
[ 1662.885769] [<8002fb15>] (smpboot_thread_fn+0x175/0x188) from [<8002a945>] (kthread+0x5d/0x6c)
[ 1662.885784] [<8002a945>] (kthread+0x5d/0x6c) from [<8000c339>] (ret_from_fork+0x11/0x38)
[ 1662.893114] ---[ end trace 0000000000000002 ]---
[ 1682.685516] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=489, c=488, q=1155)
[ 1682.685520] INFO: Stall ended before state dump startThanks & RegardsMeenakshi Aggarwal
Kernelnewbies mailing list
Kernelnewbies@xxxxxxxxxxxxxxxxx
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
--
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@xxxxxxxxxxxxxxxxx http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
