Hi all,
I have updated my Centos 6.5 KVM host to kernel 2.6.32-431.23.3 this
morning ... After 2 hours working, the following kernel error appears
and all vm guests goes slowly ....
device monif4 entered promiscuous mode
monif: port 5(monif4) entering forwarding state
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup D 0000000000000009 0 83 2 0x00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
[<ffffffff8152a20b>] mutex_lock+0x2b/0x50
[<ffffffff810c97b5>] cgroup_lock+0x15/0x20
[<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
[<ffffffff81094a20>] worker_thread+0x170/0x2a0
[<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
Uhmm , how can I debug this?? Any idea??
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
