Re: kswapd hung tasks in 2.6.38-rc1

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, 2011-02-01 at 00:13 +0100, Andrea Arcangeli wrote:
> On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote:
> > Still not a very good data point, but I ran a heavy swap load for an
> > hour or so without reproducing this.  But, it happened again after I
> > enabled transparent huge pages.  I managed to get a sysrq-t dump out of
> > it:
> > 
> > 	http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt
> > 
> > khugepaged is one of the three running tasks.  Note, I set both its
> > sleep timeouts to zero to stress it out a bit.
> 
> sysrq+l? sysrq+t doesn't provide interesting info for running tasks.

Got one again.  Took ~3 hours to reproduce this time, but I don't think
I was hitting swap very hard most of the time.

Here's sysrq+l pasted.  sysrq-t is here:

	http://sr71.net/~dave/ibm/2.6.38-rc2-hang-1.txt

[ 9834.262374] SysRq : Show backtrace of all active CPUs
[ 9834.267437] CPU0:
[ 9834.269360] CPU 0 
[ 9834.271197] Modules linked in:
[ 9834.274455] 
[ 9834.275948] Pid: 0, comm: swapper Not tainted 2.6.38-rc2-dirty #119 49Y6512     /System x3650 M2 -[7947AC1]-
[ 9834.285748] RIP: 0010:[<ffffffff81348d3d>]  [<ffffffff81348d3d>] acpi_idle_enter_bm+0x22e/0x265
[ 9834.294443] RSP: 0000:ffffffff818c9ec8  EFLAGS: 00000246
[ 9834.299742] RAX: 00000000000009db RBX: ffffffff8109dbce RCX: 00000000000003e8
[ 9834.306857] RDX: 0000000000000119 RSI: 0000000000000001 RDI: 0000000000000000
[ 9834.313971] RBP: ffffffff818c9ef8 R08: 0000000000000001 R09: 0000000000000020
[ 9834.321086] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff815f1ece
[ 9834.328201] R13: ffff88083e786800 R14: ffff88083e786820 R15: ffff88083e78680c
[ 9834.335317] FS:  0000000000000000(0000) GS:ffff88007a800000(0000) knlGS:0000000000000000
[ 9834.343386] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 9834.349116] CR2: 00007f9b2eaf5720 CR3: 000000000191b000 CR4: 00000000000006f0
[ 9834.356231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9834.363346] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 9834.370461] Process swapper (pid: 0, threadinfo ffffffff818c8000, task ffffffff81923020)
[ 9834.378530] Stack:
[ 9834.380540]  ffffffff818c9ef8 00000000814f76b6 ffff88083e786820 ffff88083e7868f0
[ 9834.387992]  0000000000000000 0000000000000002 ffffffff818c9f38 ffffffff814f68a9
[ 9834.395442]  ffffffff818c9f18 ffffffff819aaf80 6db6db6db6db6db7 000000000000001f
[ 9834.402893] Call Trace:
[ 9834.405338]  [<ffffffff814f68a9>] cpuidle_idle_call+0xb9/0x210
[ 9834.411159]  [<ffffffff81039c87>] cpu_idle+0x47/0x80
[ 9834.416113]  [<ffffffff815c61f2>] rest_init+0x72/0x80
[ 9834.421152]  [<ffffffff819c3dff>] start_kernel+0x2ff/0x3e0
[ 9834.426626]  [<ffffffff819c3271>] x86_64_start_reservations+0x81/0xd0
[ 9834.433049]  [<ffffffff819c339d>] x86_64_start_kernel+0xdd/0x100
[ 9834.439041] Code: 89 d5 ff 48 89 c6 ba e8 03 00 00 48 29 de 48 89 d1 31 d2 48 89 f0 48 89 f7 48 f7 f1 48 89 c3 e8 ba 5e d5 ff fb 41 80 7c 24 08 01 <74> 10 65 48 8b 04 25 08 b7 00 00 83 88 3c e0 ff ff 04 41 ff 44 
[ 9834.459137] Call Trace:
[ 9834.461580]  [<ffffffff814f68a9>] cpuidle_idle_call+0xb9/0x210
[ 9834.467399]  [<ffffffff81039c87>] cpu_idle+0x47/0x80
[ 9834.472353]  [<ffffffff815c61f2>] rest_init+0x72/0x80
[ 9834.477391]  [<ffffffff819c3dff>] start_kernel+0x2ff/0x3e0
[ 9834.482864]  [<ffffffff819c3271>] x86_64_start_reservations+0x81/0xd0
[ 9834.489287]  [<ffffffff819c339d>] x86_64_start_kernel+0xdd/0x100
[ 9834.495337] CPU4:
[ 9834.497266]  ffff88107fc03ef8 0000000000000046 0000000000000004 ffffffff8135edc0
[ 9834.504721]  ffff88087fffc2d8 ffff88083e067bd0 ffff88107fc03f38 ffffffff8103e74a
[ 9834.512175]  ffff88107fc03f58 ffffffff8135ee0d ffff88107fc03f68 ffff88007a812580
[ 9834.519633] Call Trace:
[ 9834.522077]  <IRQ>  [<ffffffff8135edc0>] ? showacpu+0x0/0x70
[ 9834.527747]  [<ffffffff8103e74a>] ? show_stack+0x1a/0x20
[ 9834.533049]  [<ffffffff8135ee0d>] ? showacpu+0x4d/0x70
[ 9834.538179]  [<ffffffff810aabcb>] ? generic_smp_call_function_interrupt+0x7b/0x170
[ 9834.545733]  [<ffffffff81053c27>] ? smp_call_function_interrupt+0x27/0x40
[ 9834.552505]  [<ffffffff8103b7d3>] ? call_function_interrupt+0x13/0x20
[ 9834.558927]  <EOI>  [<ffffffff810f99cd>] ? shrink_zone+0x16d/0x4c0
[ 9834.565113]  [<ffffffff810fa9db>] ? do_try_to_free_pages+0x9b/0x300
[ 9834.571365]  [<ffffffff810fae00>] ? try_to_free_pages+0x90/0x110
[ 9834.577360]  [<ffffffff810f02f5>] ? __alloc_pages_nodemask+0x355/0x730
[ 9834.583873]  [<ffffffff815f018e>] ? schedule_timeout+0x11e/0x250
[ 9834.589866]  [<ffffffff811223e2>] ? alloc_pages_vma+0x82/0x130
[ 9834.595687]  [<ffffffff8112de30>] ? khugepaged+0x8c0/0x10c0
[ 9834.601249]  [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40
[ 9834.607673]  [<ffffffff8112d570>] ? khugepaged+0x0/0x10c0
[ 9834.613059]  [<ffffffff81098c77>] ? kthread+0x97/0xa0
[ 9834.618099]  [<ffffffff8103ba54>] ? kernel_thread_helper+0x4/0x10
[ 9834.624176]  [<ffffffff81098be0>] ? kthread+0x0/0xa0
[ 9834.629132]  [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10

And the hung task messages:

[ 9687.185858] INFO: task kswapd0:706 blocked for more than 120 seconds.
[ 9687.192291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9687.200102] kswapd0         D 000000010022f817     0   706      2 0x00000000
[ 9687.207173]  ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40
[ 9687.214627]  ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001
[ 9687.222080]  ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308
[ 9687.229532] Call Trace:
[ 9687.231986]  [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120
[ 9687.237634]  [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0
[ 9687.243201]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9687.248421]  [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170
[ 9687.254154]  [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40
[ 9687.260581]  [<ffffffff812d8927>] ? elv_merge+0x47/0x1d0
[ 9687.265882]  [<ffffffff812df53e>] __make_request+0x6e/0x480
[ 9687.271446]  [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0
[ 9687.277612]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9687.282741]  [<ffffffff812df407>] submit_bio+0x67/0xe0
[ 9687.287869]  [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180
[ 9687.294386]  [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0
[ 9687.299860]  [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800
[ 9687.305682]  [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0
[ 9687.311849]  [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0
[ 9687.317239]  [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0
[ 9687.323579]  [<ffffffff810fa575>] kswapd+0x605/0x8a0
[ 9687.328534]  [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0
[ 9687.333488]  [<ffffffff81098c77>] kthread+0x97/0xa0
[ 9687.338362]  [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10
[ 9687.344269]  [<ffffffff81098be0>] ? kthread+0x0/0xa0
[ 9687.349225]  [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10
[ 9687.355303] INFO: task kswapd1:707 blocked for more than 120 seconds.
[ 9687.361725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9687.369537] kswapd1         D 000000010022f812     0   707      2 0x00000000
[ 9687.376609]  ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40
[ 9687.384062]  ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c
[ 9687.391518]  ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308
[ 9687.398971] Call Trace:
[ 9687.401414]  [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120
[ 9687.407062]  [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0
[ 9687.412625]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9687.417842]  [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170
[ 9687.423576]  [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40
[ 9687.430002]  [<ffffffff812d8a5b>] ? elv_merge+0x17b/0x1d0
[ 9687.435392]  [<ffffffff812df53e>] __make_request+0x6e/0x480
[ 9687.440952]  [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0
[ 9687.447121]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9687.452251]  [<ffffffff812df407>] submit_bio+0x67/0xe0
[ 9687.457377]  [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180
[ 9687.463889]  [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0
[ 9687.469364]  [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800
[ 9687.475183]  [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0
[ 9687.481349]  [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0
[ 9687.486741]  [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0
[ 9687.493082]  [<ffffffff810fa575>] kswapd+0x605/0x8a0
[ 9687.498038]  [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0
[ 9687.502992]  [<ffffffff81098c77>] kthread+0x97/0xa0
[ 9687.507862]  [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10
[ 9687.513768]  [<ffffffff81098be0>] ? kthread+0x0/0xa0
[ 9687.518723]  [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10
[ 9687.524805] INFO: task memknobs:2741 blocked for more than 120 seconds.
[ 9687.531403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9687.539211] memknobs        D 000000010022f7f5     0  2741   2674 0x00000000
[ 9687.546282]  ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40
[ 9687.553735]  ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e
[ 9687.561189]  0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2
[ 9687.568644] Call Trace:
[ 9687.571089]  [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0
[ 9687.577429]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9687.582556]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9687.587769]  [<ffffffff810e9525>] sync_page+0x45/0x60
[ 9687.592811]  [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90
[ 9687.598197]  [<ffffffff810e94e0>] ? sync_page+0x0/0x60
[ 9687.603324]  [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80
[ 9687.608973]  [<ffffffff81099190>] ? wake_bit_function+0x0/0x40
[ 9687.614795]  [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10
[ 9687.621048]  [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60
[ 9687.627043]  [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0
[ 9687.632864]  [<ffffffff810ef167>] ? __free_pages+0x27/0x30
[ 9687.638340]  [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110
[ 9687.643813]  [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270
[ 9687.649547]  [<ffffffff815f4eb1>] do_page_fault+0x161/0x480
[ 9687.655108]  [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0
[ 9687.660758]  [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0
[ 9687.666578]  [<ffffffff815f20df>] page_fault+0x1f/0x30
[ 9687.671708] INFO: task memknobs:2794 blocked for more than 120 seconds.
[ 9687.678308] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9687.686117] memknobs        D 000000010022f90a     0  2794   2724 0x00000000
[ 9687.693187]  ffff880c66627c68 0000000000000082 ffff88083faa8f40 0000000000011c40
[ 9687.700639]  ffff88103e26cae0 ffff88103e26c780 ffffffff81923020 ffff880800000000
[ 9687.708095]  0000000000000008 0000000000821ba7 ffff880c66627c38 ffffffff812db5d2
[ 9687.715551] Call Trace:
[ 9687.717998]  [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0
[ 9687.724338]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9687.729555]  [<ffffffff810e9525>] sync_page+0x45/0x60
[ 9687.734597]  [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90
[ 9687.739984]  [<ffffffff810e94e0>] ? sync_page+0x0/0x60
[ 9687.745112]  [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80
[ 9687.750759]  [<ffffffff81099190>] ? wake_bit_function+0x0/0x40
[ 9687.756578]  [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10
[ 9687.762830]  [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60
[ 9687.768824]  [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0
[ 9687.774645]  [<ffffffff810ef167>] ? __free_pages+0x27/0x30
[ 9687.780118]  [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110
[ 9687.785591]  [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270
[ 9687.791324]  [<ffffffff815f4eb1>] do_page_fault+0x161/0x480
[ 9687.796884]  [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0
[ 9687.802530]  [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0
[ 9687.808351]  [<ffffffff815f20df>] page_fault+0x1f/0x30
[ 9807.612419] INFO: task kswapd0:706 blocked for more than 120 seconds.
[ 9807.618845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9807.626656] kswapd0         D 000000010022f817     0   706      2 0x00000000
[ 9807.633728]  ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40
[ 9807.641187]  ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001
[ 9807.648640]  ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308
[ 9807.656097] Call Trace:
[ 9807.658544]  [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120
[ 9807.664190]  [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0
[ 9807.669754]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9807.674969]  [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170
[ 9807.680705]  [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40
[ 9807.687131]  [<ffffffff812d8927>] ? elv_merge+0x47/0x1d0
[ 9807.692432]  [<ffffffff812df53e>] __make_request+0x6e/0x480
[ 9807.697995]  [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0
[ 9807.704160]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9807.709288]  [<ffffffff812df407>] submit_bio+0x67/0xe0
[ 9807.714417]  [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180
[ 9807.720932]  [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0
[ 9807.726406]  [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800
[ 9807.732231]  [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0
[ 9807.738396]  [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0
[ 9807.743788]  [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0
[ 9807.750127]  [<ffffffff810fa575>] kswapd+0x605/0x8a0
[ 9807.755081]  [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0
[ 9807.760037]  [<ffffffff81098c77>] kthread+0x97/0xa0
[ 9807.764908]  [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10
[ 9807.770814]  [<ffffffff81098be0>] ? kthread+0x0/0xa0
[ 9807.775770]  [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10
[ 9807.781851] INFO: task kswapd1:707 blocked for more than 120 seconds.
[ 9807.788277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9807.796084] kswapd1         D 000000010022f812     0   707      2 0x00000000
[ 9807.803158]  ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40
[ 9807.810615]  ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c
[ 9807.818070]  ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308
[ 9807.825524] Call Trace:
[ 9807.827967]  [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120
[ 9807.833614]  [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0
[ 9807.839176]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9807.844393]  [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170
[ 9807.850126]  [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40
[ 9807.856551]  [<ffffffff812d8a5b>] ? elv_merge+0x17b/0x1d0
[ 9807.861940]  [<ffffffff812df53e>] __make_request+0x6e/0x480
[ 9807.867503]  [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0
[ 9807.873669]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9807.878796]  [<ffffffff812df407>] submit_bio+0x67/0xe0
[ 9807.883926]  [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180
[ 9807.890439]  [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0
[ 9807.895915]  [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800
[ 9807.901737]  [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0
[ 9807.907905]  [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0
[ 9807.913335]  [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0
[ 9807.919675]  [<ffffffff810fa575>] kswapd+0x605/0x8a0
[ 9807.924632]  [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0
[ 9807.929588]  [<ffffffff81098c77>] kthread+0x97/0xa0
[ 9807.934457]  [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10
[ 9807.940365]  [<ffffffff81098be0>] ? kthread+0x0/0xa0
[ 9807.945321]  [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10
[ 9807.951404] INFO: task avahi-daemon:2353 blocked for more than 120 seconds.
[ 9807.958346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9807.966155] avahi-daemon    D 00000001002332a4     0  2353      1 0x00000000
[ 9807.973227]  ffff88103f5437a8 0000000000000086 ffff88083faa8f40 0000000000011c40
[ 9807.980682]  ffff88103eb764e0 ffff88103eb76180 ffffffff81923020 ffff880800000000
[ 9807.988134]  0000000000000008 00000000007e89ff ffff88103f543778 ffffffff812db5d2
[ 9807.995587] Call Trace:
[ 9807.998034]  [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0
[ 9808.004374]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9808.009501]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9808.014716]  [<ffffffff810e9525>] sync_page+0x45/0x60
[ 9808.019759]  [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90
[ 9808.025146]  [<ffffffff810e94e0>] ? sync_page+0x0/0x60
[ 9808.030273]  [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80
[ 9808.035922]  [<ffffffff81099190>] ? wake_bit_function+0x0/0x40
[ 9808.041745]  [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10
[ 9808.047999]  [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60
[ 9808.053994]  [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0
[ 9808.059814]  [<ffffffff810ef167>] ? __free_pages+0x27/0x30
[ 9808.065290]  [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110
[ 9808.070766]  [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270
[ 9808.076500]  [<ffffffff815f4eb1>] do_page_fault+0x161/0x480
[ 9808.082063]  [<ffffffff812f4eba>] ? rb_insert_color+0x8a/0xf0
[ 9808.087800]  [<ffffffff812f64b0>] ? timerqueue_add+0x60/0xb0
[ 9808.093449]  [<ffffffff8109c891>] ? lock_hrtimer_base+0x31/0x60
[ 9808.099356]  [<ffffffff8109c961>] ? hrtimer_try_to_cancel+0x51/0xd0
[ 9808.105610]  [<ffffffff815f20df>] page_fault+0x1f/0x30
[ 9808.110741]  [<ffffffff8114848c>] ? do_sys_poll+0x3ac/0x460
[ 9808.116301]  [<ffffffff81148450>] ? do_sys_poll+0x370/0x460
[ 9808.121863]  [<ffffffff81147e80>] ? __pollwait+0x0/0x100
[ 9808.127164]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.132206]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.137248]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.142291]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.147331]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.152373]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.157416]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.162457]  [<ffffffff81147f80>] ? pollwake+0x0/0x70
[ 9808.167530]  [<ffffffff81137365>] ? do_sync_read+0xd5/0x120
[ 9808.173093]  [<ffffffff8107fb39>] ? timespec_add_safe+0x39/0x70
[ 9808.179000]  [<ffffffff81147d34>] ? poll_select_set_timeout+0x84/0xa0
[ 9808.185426]  [<ffffffff811485bc>] sys_poll+0x7c/0x100
[ 9808.190469]  [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b
[ 9808.196463] INFO: task rsyslogd:2523 blocked for more than 120 seconds.
[ 9808.203061] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9808.210872] rsyslogd        D 0000000100237de3     0  2523      1 0x00000000
[ 9808.217944]  ffff88083ec11a88 0000000000000086 ffff88083faa8f40 0000000000011c40
[ 9808.225403]  ffff88083f93a460 ffff88083f93a100 ffffffff81923020 ffff880800000000
[ 9808.232858]  0000000000000008 00000000006f52cf ffff88083ec11a58 ffffffff812db5d2
[ 9808.240311] Call Trace:
[ 9808.242754]  [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0
[ 9808.249092]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9808.254220]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9808.259571]  [<ffffffff810e9525>] sync_page+0x45/0x60
[ 9808.264616]  [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90
[ 9808.270002]  [<ffffffff810e94e0>] ? sync_page+0x0/0x60
[ 9808.275131]  [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80
[ 9808.280780]  [<ffffffff81099190>] ? wake_bit_function+0x0/0x40
[ 9808.286600]  [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10
[ 9808.292855]  [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60
[ 9808.298849]  [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0
[ 9808.304668]  [<ffffffff810ef167>] ? __free_pages+0x27/0x30
[ 9808.310142]  [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110
[ 9808.315618]  [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270
[ 9808.321352]  [<ffffffff815f4eb1>] do_page_fault+0x161/0x480
[ 9808.326913]  [<ffffffff81039996>] ? __switch_to+0x1e6/0x2a0
[ 9808.332472]  [<ffffffff815ef808>] ? schedule+0x3f8/0xaf0
[ 9808.337775]  [<ffffffff815f20df>] page_fault+0x1f/0x30
[ 9808.342904]  [<ffffffff8107bd47>] ? do_syslog+0x347/0x4e0
[ 9808.348292]  [<ffffffff8107bd02>] ? do_syslog+0x302/0x4e0
[ 9808.353681]  [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40
[ 9808.360107]  [<ffffffff81193f36>] kmsg_read+0x36/0x70
[ 9808.365149]  [<ffffffff8118a324>] proc_reg_read+0x74/0xb0
[ 9808.370538]  [<ffffffff81137b7b>] vfs_read+0xcb/0x170
[ 9808.375581]  [<ffffffff81138025>] sys_read+0x55/0x90
[ 9808.380539]  [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b
[ 9808.386532] INFO: task memknobs:2741 blocked for more than 120 seconds.
[ 9808.393133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9808.400942] memknobs        D 000000010022f7f5     0  2741   2674 0x00000000
[ 9808.408014]  ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40
[ 9808.415471]  ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e
[ 9808.422926]  0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2
[ 9808.430377] Call Trace:
[ 9808.432824]  [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0
[ 9808.439165]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9808.444295]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9808.449511]  [<ffffffff810e9525>] sync_page+0x45/0x60
[ 9808.454553]  [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90
[ 9808.459943]  [<ffffffff810e94e0>] ? sync_page+0x0/0x60
[ 9808.465070]  [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80
[ 9808.470715]  [<ffffffff81099190>] ? wake_bit_function+0x0/0x40
[ 9808.476535]  [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10
[ 9808.482788]  [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60
[ 9808.488781]  [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0
[ 9808.494604]  [<ffffffff810ef167>] ? __free_pages+0x27/0x30
[ 9808.500077]  [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110
[ 9808.505549]  [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270
[ 9808.511284]  [<ffffffff815f4eb1>] do_page_fault+0x161/0x480
[ 9808.516848]  [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0
[ 9808.522498]  [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0
[ 9808.528318]  [<ffffffff815f20df>] page_fault+0x1f/0x30
[ 9808.533449] INFO: task sshd:2755 blocked for more than 120 seconds.
[ 9808.539700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9808.547511] sshd            D 00000001002310ff     0  2755   2510 0x00000000
[ 9808.554582]  ffff880079c27c68 0000000000000086 ffff88083faa8f40 0000000000011c40
[ 9808.562039]  ffff88083e2f1aa0 ffff88083e2f1740 ffffffff81923020 ffff880800000000
[ 9808.569493]  0000000000000008 00000000007de1b7 ffff880079c27c38 ffffffff812db5d2
[ 9808.576948] Call Trace:
[ 9808.579393]  [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0
[ 9808.585731]  [<ffffffff811646ed>] ? bio_init+0x1d/0x40
[ 9808.590859]  [<ffffffff815effe4>] io_schedule+0x44/0x60
[ 9808.596075]  [<ffffffff810e9525>] sync_page+0x45/0x60
[ 9808.601115]  [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90
[ 9808.606502]  [<ffffffff810e94e0>] ? sync_page+0x0/0x60
[ 9808.611630]  [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80
[ 9808.617277]  [<ffffffff81099190>] ? wake_bit_function+0x0/0x40
[ 9808.623096]  [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10
[ 9808.629350]  [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60
[ 9808.635345]  [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0
[ 9808.641166]  [<ffffffff810ef167>] ? __free_pages+0x27/0x30
[ 9808.646643]  [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110
[ 9808.652118]  [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270
[ 9808.657851]  [<ffffffff815f4eb1>] do_page_fault+0x161/0x480
[ 9808.663413]  [<ffffffff81137bde>] ? vfs_read+0x12e/0x170
[ 9808.668713]  [<ffffffff815f20df>] page_fault+0x1f/0x30


-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxxx  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]