kswapd hung tasks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is not always reproducible so far but I thought it would send it out anyway in case someone could spot the problem. Running some memory allocation using cpuset with swapping workloads (like oom02 in LTP) caused hung tasks in 2.6.38-rc2 kernel.

Is this something that it took a long time to move memory into the swap device?

Would this go away by replacing with a fast storage for swapping?

INFO: task kswapd1:275 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kswapd1         D ffff88045dd8c5a0     0   275      2 0x00000000
 ffff88045dda5810 0000000000000046 0000000000011210 ffff880500000000
 0000000000014d40 ffff88045dd8c040 ffff88045dd8c5a0 ffff88045dda5fd8
 ffff88045dd8c5a8 0000000000014d40 ffff88045dda4010 0000000000014d40
Call Trace:
 [<ffffffff814a2330>] io_schedule+0x70/0xc0
 [<ffffffff8120463a>] get_request_wait+0xca/0x1a0
 [<ffffffff8107fd50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811fd197>] ? elv_merge+0x1d7/0x210
 [<ffffffff8120477b>] __make_request+0x6b/0x4d0
 [<ffffffff8100cb2e>] ? call_function_interrupt+0xe/0x20
 [<ffffffff8120259f>] generic_make_request+0x21f/0x5b0
 [<ffffffff810f7885>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff810f7a23>] ? mempool_alloc+0x63/0x150
 [<ffffffff812029b6>] submit_bio+0x86/0x110
 [<ffffffff810ffb96>] ? test_set_page_writeback+0x106/0x190
 [<ffffffff81127f63>] swap_writepage+0x83/0xd0
 [<ffffffff811048fe>] pageout+0x12e/0x310
 [<ffffffff81104efa>] shrink_page_list+0x41a/0x5a0
 [<ffffffff81105686>] shrink_inactive_list+0x166/0x480
 [<ffffffff81106023>] shrink_zone+0x363/0x4d0
 [<ffffffff810fa4e5>] ? zone_watermark_ok_safe+0xb5/0xd0
 [<ffffffff811077a9>] kswapd+0x949/0xbf0
 [<ffffffff81106e60>] ? kswapd+0x0/0xbf0
 [<ffffffff8107f6c6>] kthread+0x96/0xa0
 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8107f630>] ? kthread+0x0/0xa0
 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10
INFO: task kswapd2:276 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kswapd2         D ffff88045dda79e0     0   276      2 0x00000000
 ffff88045dda9810 0000000000000046 0000000000011210 ffff880b00000000
 0000000000014d40 ffff88045dda7480 ffff88045dda79e0 ffff88045dda9fd8
 ffff88045dda79e8 0000000000014d40 ffff88045dda8010 0000000000014d40
Call Trace:
 [<ffffffff814a2330>] io_schedule+0x70/0xc0
 [<ffffffff8120463a>] get_request_wait+0xca/0x1a0
 [<ffffffff8107fd50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811fd197>] ? elv_merge+0x1d7/0x210
 [<ffffffff8120477b>] __make_request+0x6b/0x4d0
 [<ffffffff8120259f>] generic_make_request+0x21f/0x5b0
 [<ffffffff810f7885>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff810f7a23>] ? mempool_alloc+0x63/0x150
 [<ffffffff812029b6>] submit_bio+0x86/0x110
 [<ffffffff810ffb96>] ? test_set_page_writeback+0x106/0x190
 [<ffffffff81127f63>] swap_writepage+0x83/0xd0
 [<ffffffff811048fe>] pageout+0x12e/0x310
 [<ffffffff81104efa>] shrink_page_list+0x41a/0x5a0
 [<ffffffff81105686>] shrink_inactive_list+0x166/0x480
 [<ffffffff810fe7aa>] ? determine_dirtyable_memory+0x1a/0x30
 [<ffffffff81106023>] shrink_zone+0x363/0x4d0
 [<ffffffff810fa4e5>] ? zone_watermark_ok_safe+0xb5/0xd0
 [<ffffffff811077a9>] kswapd+0x949/0xbf0
 [<ffffffff81106e60>] ? kswapd+0x0/0xbf0
 [<ffffffff8107f6c6>] kthread+0x96/0xa0
 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8107f630>] ? kthread+0x0/0xa0
 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10
INFO: task jbd2/dm-0-8:1180 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/dm-0-8     D ffff88045df370e0     0  1180      2 0x00000000
 ffff88045d34f430 0000000000000046 0000000000011210 ffff880b00000000
 0000000000014d40 ffff88045df36b80 ffff88045df370e0 ffff88045d34ffd8
 ffff88045df370e8 0000000000014d40 ffff88045d34e010 0000000000014d40
Call Trace:
 [<ffffffff814a2330>] io_schedule+0x70/0xc0
 [<ffffffff8120463a>] get_request_wait+0xca/0x1a0
 [<ffffffff8107fd50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811fd007>] ? elv_merge+0x47/0x210
 [<ffffffff8120477b>] __make_request+0x6b/0x4d0
 [<ffffffff8120259f>] generic_make_request+0x21f/0x5b0
 [<ffffffff810f7885>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff810f7a23>] ? mempool_alloc+0x63/0x150
 [<ffffffff8100c96e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff812029b6>] submit_bio+0x86/0x110
 [<ffffffff810ffb96>] ? test_set_page_writeback+0x106/0x190
 [<ffffffff81127f63>] swap_writepage+0x83/0xd0
 [<ffffffff811048fe>] pageout+0x12e/0x310
 [<ffffffff81104efa>] shrink_page_list+0x41a/0x5a0
 [<ffffffff81105686>] shrink_inactive_list+0x166/0x480
 [<ffffffff8104c846>] ? enqueue_task+0x66/0x80
 [<ffffffff810fe7aa>] ? determine_dirtyable_memory+0x1a/0x30
 [<ffffffff81106023>] shrink_zone+0x363/0x4d0
 [<ffffffff8110470e>] ? shrink_slab+0x14e/0x180
 [<ffffffff8110675f>] do_try_to_free_pages+0xaf/0x4a0
 [<ffffffff81106dc2>] try_to_free_pages+0x92/0x130
 [<ffffffff8104c643>] ? __wake_up+0x53/0x70
 [<ffffffff810fc46b>] __alloc_pages_nodemask+0x3fb/0x760
 [<ffffffff8113349a>] alloc_pages_current+0x9a/0x100
 [<ffffffff810f55b7>] __page_cache_alloc+0x87/0x90
 [<ffffffff810f61ff>] find_or_create_page+0x4f/0xb0
 [<ffffffff8117575c>] __getblk+0xec/0x330
 [<ffffffffa0089bae>] jbd2_journal_get_descriptor_buffer+0x4e/0xb0 [jbd2]
 [<ffffffffa008307d>] jbd2_journal_commit_transaction+0x99d/0x13f0 [jbd2]
 [<ffffffff8106e2a3>] ? try_to_del_timer_sync+0x83/0xe0
 [<ffffffffa00882c8>] kjournald2+0xb8/0x220 [jbd2]
 [<ffffffff8107fd50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa0088210>] ? kjournald2+0x0/0x220 [jbd2]
 [<ffffffff8107f6c6>] kthread+0x96/0xa0
 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8107f630>] ? kthread+0x0/0xa0
 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10
INFO: task irqbalance:3426 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
irqbalance      D ffff880c5e96c5a0     0  3426      1 0x00000080
 ffff880c5e911318 0000000000000086 0000000000011210 ffff881027b470c0
 0000000000014d40 ffff880c5e96c040 ffff880c5e96c5a0 ffff880c5e911fd8
 ffff880c5e96c5a8 0000000000014d40 ffff880c5e910010 0000000000014d40
Call Trace:
 [<ffffffff814a2330>] io_schedule+0x70/0xc0
 [<ffffffff8120463a>] get_request_wait+0xca/0x1a0
 [<ffffffff8107fd50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811fd197>] ? elv_merge+0x1d7/0x210
 [<ffffffff8120477b>] __make_request+0x6b/0x4d0
 [<ffffffff8120259f>] generic_make_request+0x21f/0x5b0
 [<ffffffff810f7885>] ? mempool_alloc_slab+0x15/0x20
 [<ffffffff810f7a23>] ? mempool_alloc+0x63/0x150
 [<ffffffff812029b6>] submit_bio+0x86/0x110
 [<ffffffff810ffb96>] ? test_set_page_writeback+0x106/0x190
 [<ffffffff81127f63>] swap_writepage+0x83/0xd0
 [<ffffffff811048fe>] pageout+0x12e/0x310
 [<ffffffff81104efa>] shrink_page_list+0x41a/0x5a0
 [<ffffffff81105686>] shrink_inactive_list+0x166/0x480
 [<ffffffff812198a9>] ? cpumask_next_and+0x29/0x50
 [<ffffffff8104d535>] ? select_idle_sibling+0x95/0x150
 [<ffffffff810fe7aa>] ? determine_dirtyable_memory+0x1a/0x30
 [<ffffffff81106023>] shrink_zone+0x363/0x4d0
 [<ffffffff810899f3>] ? ktime_get_ts+0xb3/0xf0
 [<ffffffff8110675f>] do_try_to_free_pages+0xaf/0x4a0
 [<ffffffff81106dc2>] try_to_free_pages+0x92/0x130
 [<ffffffff8104c643>] ? __wake_up+0x53/0x70
 [<ffffffff810fc46b>] __alloc_pages_nodemask+0x3fb/0x760
 [<ffffffff8113349a>] alloc_pages_current+0x9a/0x100
 [<ffffffff810fb6ce>] __get_free_pages+0xe/0x50
 [<ffffffff811dacd4>] inode_doinit_with_dentry+0x144/0x670
 [<ffffffff8107fc17>] ? bit_waitqueue+0x17/0xd0
 [<ffffffff811db21c>] selinux_d_instantiate+0x1c/0x20
 [<ffffffff811d2bfb>] security_d_instantiate+0x1b/0x30
 [<ffffffff8115da0a>] d_instantiate+0x4a/0x70
 [<ffffffff811a9a90>] proc_lookup_de+0xa0/0x100
 [<ffffffff811a9b0b>] proc_lookup+0x1b/0x20
 [<ffffffff811a3b27>] proc_root_lookup+0x27/0x50
 [<ffffffff81153d15>] d_alloc_and_lookup+0x45/0x90
 [<ffffffff8115e3f5>] ? d_lookup+0x35/0x60
 [<ffffffff81156701>] do_lookup+0x1e1/0x2c0
 [<ffffffff81156d5b>] link_path_walk+0x57b/0xa90
 [<ffffffff8115754b>] do_path_lookup+0x5b/0x130
 [<ffffffff811585e7>] do_filp_open+0x1f7/0x770
 [<ffffffff811391ee>] ? fallback_alloc+0x14e/0x270
 [<ffffffff81164105>] ? alloc_fd+0x95/0x160
 [<ffffffff81147669>] do_sys_open+0x69/0x110
 [<ffffffff81147750>] sys_open+0x20/0x30
 [<ffffffff8100bf82>] system_call_fastpath+0x16/0x1b

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxxx  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]