Or these:
359.933112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 359.949567] ceph-osd D ffffffff8180e9c0 0 3841 1
0x00000000
[ 359.966452] ffff880db567d7b8 0000000000000082 ffff880db6c30000
0000000000012200
[ 359.983845] ffff880db567dfd8 ffff880db567c010 0000000000012200
0000000000012200
[ 360.001308] ffff880db567dfd8 0000000000012200 ffff881038ee1810
ffff880db6c30000
[ 360.019065] Call Trace:
[ 360.036550] [<ffffffff816296e4>] schedule+0x24/0x70
[ 360.054562] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[ 360.054566] [<ffffffff812affe4>] ? xfs_iext_bno_to_ext+0x84/0x160
[ 360.054568] [<ffffffff8162888a>] __down+0x6a/0xb0
[ 360.054571] [<ffffffff8126d885>] ? _xfs_buf_find+0xe5/0x230
[ 360.054573] [<ffffffff8106310b>] down+0x3b/0x50
[ 360.054575] [<ffffffff8126d640>] xfs_buf_lock+0x40/0xe0
[ 360.054576] [<ffffffff8126d885>] _xfs_buf_find+0xe5/0x230
[ 360.054578] [<ffffffff8126d9ff>] xfs_buf_get+0x2f/0x1b0
[ 360.054580] [<ffffffff812bb2a2>] ? xfs_icsb_modify_counters+0xa2/0x190
[ 360.054581] [<ffffffff8126e1f7>] xfs_buf_read+0x27/0x100
[ 360.054584] [<ffffffff812c64c7>] xfs_trans_read_buf+0x1f7/0x430
[ 360.054588] [<ffffffff81283ab1>] xfs_read_agf+0x61/0x190
[ 360.054589] [<ffffffff81283c14>] xfs_alloc_read_agf+0x34/0xd0
[ 360.054592] [<ffffffff81286fd7>] xfs_alloc_fix_freelist+0x427/0x4a0
[ 360.054594] [<ffffffff816296e4>] ? schedule+0x24/0x70
[ 360.054596] [<ffffffff8162a21d>] ? rwsem_down_failed_common+0xbd/0x150
[ 360.054599] [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
[ 360.054601] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[ 360.054603] [<ffffffff812bab63>] ? xfs_perag_get+0x33/0xb0
[ 360.054605] [<ffffffff812870e7>] ? xfs_free_extent+0x97/0x140
[ 360.054607] [<ffffffff812870fb>] xfs_free_extent+0xab/0x140
[ 360.054609] [<ffffffff812915bd>] xfs_bmap_finish+0x15d/0x1a0
[ 360.054611] [<ffffffff812b3487>] xfs_itruncate_extents+0xf7/0x2a0
[ 360.054613] [<ffffffff81278d34>] xfs_setattr_size+0x394/0x410
[ 360.054615] [<ffffffff81278de2>] xfs_vn_setattr+0x32/0x40
[ 360.054617] [<ffffffff8112481a>] notify_change+0x13a/0x2e0
[ 360.054619] [<ffffffff8110a28f>] do_truncate+0x5f/0x90
[ 360.054621] [<ffffffff8110a4bf>] sys_truncate+0x12f/0x150
[ 360.054623] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
[ 360.054625] INFO: task ceph-osd:3843 blocked for more than 120 seconds.
[ 360.054626] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 360.054627] ceph-osd D 0000000000000002 0 3843 1
0x00000000
[ 360.054629] ffff880db5689678 0000000000000082 ffff880db5681810
0000000000012200
[ 360.054631] ffff880db5689fd8 ffff880db5688010 0000000000012200
0000000000012200
[ 360.054633] ffff880db5689fd8 0000000000012200 ffff881038f3e040
ffff880db5681810
[ 360.054635] Call Trace:
[ 360.054637] [<ffffffff816296e4>] schedule+0x24/0x70
[ 360.054639] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[ 360.054641] [<ffffffff8106aa44>] ? T.1708+0x44/0x50
[ 360.054643] [<ffffffff8106ddaf>] ? try_to_wake_up+0x23f/0x2b0
[ 360.054645] [<ffffffff81628d54>] wait_for_common+0xc4/0x160
[ 360.054647] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[ 360.054649] [<ffffffff81628e98>] wait_for_completion+0x18/0x20
[ 360.054650] [<ffffffff81283a3d>] xfs_alloc_vextent+0x6d/0x80
[ 360.054652] [<ffffffff812922b1>] xfs_bmap_btalloc+0x2a1/0x7d0
[ 360.054654] [<ffffffff81298301>] ?
xfs_bmap_add_extent_delay_real+0xff1/0x17a0
[ 360.054656] [<ffffffff81287780>] ? __xfs_alloc_vextent+0x5f0/0x5f0
[ 360.054658] [<ffffffff812927e9>] xfs_bmap_alloc+0x9/0x10
[ 360.054659] [<ffffffff81298b78>] xfs_bmapi_allocate+0xc8/0x2c0
[ 360.054661] [<ffffffff81299279>] xfs_bmapi_write+0x509/0x780
[ 360.054664] [<ffffffff81276ab4>] xfs_iomap_write_allocate+0x134/0x370
[ 360.054666] [<ffffffff8126b325>] xfs_map_blocks+0x155/0x230
[ 360.054667] [<ffffffff8126b597>] xfs_vm_writepage+0x197/0x540
[ 360.054669] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[ 360.054672] [<ffffffff810cc4a2>] __writepage+0x12/0x40
[ 360.054674] [<ffffffff810cd5f6>] write_cache_pages+0x236/0x4c0
[ 360.054675] [<ffffffff812c27ef>] ? xfs_log_commit_cil+0x3bf/0x480
[ 360.054677] [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
[ 360.054679] [<ffffffff812bccac>] ? xfs_trans_free+0x5c/0x70
[ 360.054680] [<ffffffff812bde3b>] ? xfs_trans_commit+0xab/0x260
[ 360.054682] [<ffffffff8127362e>] ? xfs_iunlock+0x7e/0xd0
[ 360.054684] [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
[ 360.054686] [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
[ 360.054688] [<ffffffff810cd90c>] do_writepages+0x1c/0x40
[ 360.054689] [<ffffffff810c3b7b>] __filemap_fdatawrite_range+0x5b/0x60
[ 360.054691] [<ffffffff810c3e1e>] filemap_fdatawrite_range+0xe/0x10
[ 360.054694] [<ffffffff811361ae>] sys_sync_file_range+0x14e/0x180
[ 360.054696] [<ffffffff81109240>] ? filp_close+0x60/0x90
[ 360.054698] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
Am 26.06.2012 19:49, schrieb Stefan Priebe:
and these:
359.919732] INFO: task xfsaild/sdd1:2247 blocked for more than 120
seconds.
[ 359.934620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 359.950059] xfsaild/sdd1 D 0000000000000007 0 2247 2
0x00000000
[ 359.966026] ffff8810349d9cf0 0000000000000046 ffff881034809810
0000000000012200
[ 359.982249] ffff8810349d9fd8 ffff8810349d8010 0000000000012200
0000000000012200
[ 359.998634] ffff8810349d9fd8 0000000000012200 ffff881038ef8000
ffff881034809810
[ 360.015139] Call Trace:
[ 360.031552] [<ffffffff816296e4>] schedule+0x24/0x70
[ 360.048356] [<ffffffff812c2a9a>] xlog_cil_force_lsn+0xea/0x110
[ 360.065363] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[ 360.082348] [<ffffffff816279c9>] ? schedule_timeout+0x189/0x2e0
[ 360.099128] [<ffffffff812c0ab4>] _xfs_log_force+0x64/0x240
[ 360.115616] [<ffffffff812c0e13>] xfs_log_force+0x13/0x40
[ 360.132031] [<ffffffff812c5646>] xfsaild+0x106/0x6d0
[ 360.148296] [<ffffffff812c5540>] ?
xfs_trans_ail_update_bulk+0x260/0x260
[ 360.164963] [<ffffffff8105d8d6>] kthread+0x96/0xa0
[ 360.181658] [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
[ 360.198516] [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
[ 360.215534] [<ffffffff8162c090>] ? gs_change+0xb/0xb
[ 480.048246] INFO: task ceph-osd:3282 blocked for more than 120 seconds.
[ 480.065939] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 480.083979] ceph-osd D ffffffff8180e9c0 0 3282 1
0x00000004
[ 480.102327] ffff880dbbb9d7b8 0000000000000086 ffff880dbbba0000
0000000000012200
[ 480.121333] ffff880dbbb9dfd8 ffff880dbbb9c010 0000000000012200
0000000000012200
[ 480.140374] ffff880dbbb9dfd8 0000000000012200 ffff881038e69810
ffff880dbbba0000
[ 480.159738] Call Trace:
[ 480.178812] [<ffffffff816296e4>] schedule+0x24/0x70
[ 480.198230] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[ 480.217775] [<ffffffff812affe4>] ? xfs_iext_bno_to_ext+0x84/0x160
[ 480.237500] [<ffffffff8162888a>] __down+0x6a/0xb0
[ 480.257254] [<ffffffff8126d885>] ? _xfs_buf_find+0xe5/0x230
[ 480.277179] [<ffffffff8106310b>] down+0x3b/0x50
[ 480.297102] [<ffffffff8126d640>] xfs_buf_lock+0x40/0xe0
[ 480.317000] [<ffffffff8126d885>] _xfs_buf_find+0xe5/0x230
[ 480.337193] [<ffffffff8126d9ff>] xfs_buf_get+0x2f/0x1b0
[ 480.357648] [<ffffffff812bb2a2>] ? xfs_icsb_modify_counters+0xa2/0x190
[ 480.378666] [<ffffffff8126e1f7>] xfs_buf_read+0x27/0x100
[ 480.399694] [<ffffffff812c64c7>] xfs_trans_read_buf+0x1f7/0x430
[ 480.421093] [<ffffffff81283ab1>] xfs_read_agf+0x61/0x190
[ 480.442728] [<ffffffff81282e67>] ? kmem_zone_alloc+0x77/0xf0
[ 480.464717] [<ffffffff81283c14>] xfs_alloc_read_agf+0x34/0xd0
[ 480.486433] [<ffffffff81286fd7>] xfs_alloc_fix_freelist+0x427/0x4a0
[ 480.508220] [<ffffffff8129350f>] ? xfs_bmap_del_extent+0x45f/0xac0
[ 480.530335] [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
[ 480.552300] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[ 480.574439] [<ffffffff812bab63>] ? xfs_perag_get+0x33/0xb0
[ 480.596523] [<ffffffff812870e7>] ? xfs_free_extent+0x97/0x140
[ 480.618126] [<ffffffff812870fb>] xfs_free_extent+0xab/0x140
[ 480.639105] [<ffffffff812915bd>] xfs_bmap_finish+0x15d/0x1a0
[ 480.659636] [<ffffffff812b3487>] xfs_itruncate_extents+0xf7/0x2a0
[ 480.659645] [<ffffffff81278d34>] xfs_setattr_size+0x394/0x410
[ 480.659647] [<ffffffff81278de2>] xfs_vn_setattr+0x32/0x40
[ 480.659650] [<ffffffff8112481a>] notify_change+0x13a/0x2e0
[ 480.659652] [<ffffffff8110a28f>] do_truncate+0x5f/0x90
[ 480.659654] [<ffffffff8110a4bf>] sys_truncate+0x12f/0x150
[ 480.659657] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
[ 480.659659] INFO: task ceph-osd:3283 blocked for more than 120 seconds.
[ 480.659659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 480.659661] ceph-osd D 0000000000000001 0 3283 1
0x00000004
[ 480.659663] ffff880dbbb9f678 0000000000000086 ffff880dbbba1810
0000000000012200
[ 480.659666] ffff880dbbb9ffd8 ffff880dbbb9e010 0000000000012200
0000000000012200
[ 480.659669] ffff880dbbb9ffd8 0000000000012200 ffff881036ab1810
ffff880dbbba1810
[ 480.659671] Call Trace:
[ 480.659674] [<ffffffff816296e4>] schedule+0x24/0x70
[ 480.659676] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[ 480.659679] [<ffffffff8106aa44>] ? T.1708+0x44/0x50
[ 480.659681] [<ffffffff8106ddaf>] ? try_to_wake_up+0x23f/0x2b0
[ 480.659684] [<ffffffff81628d54>] wait_for_common+0xc4/0x160
[ 480.659686] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[ 480.659688] [<ffffffff81628e98>] wait_for_completion+0x18/0x20
[ 480.659691] [<ffffffff81283a3d>] xfs_alloc_vextent+0x6d/0x80
[ 480.659693] [<ffffffff812922b1>] xfs_bmap_btalloc+0x2a1/0x7d0
[ 480.659695] [<ffffffff81066215>] ? check_preempt_curr+0x75/0xa0
[ 480.659698] [<ffffffff81287780>] ? __xfs_alloc_vextent+0x5f0/0x5f0
[ 480.659700] [<ffffffff812927e9>] xfs_bmap_alloc+0x9/0x10
[ 480.659702] [<ffffffff81298b78>] xfs_bmapi_allocate+0xc8/0x2c0
[ 480.659704] [<ffffffff81299279>] xfs_bmapi_write+0x509/0x780
[ 480.659707] [<ffffffff81276ab4>] xfs_iomap_write_allocate+0x134/0x370
[ 480.659709] [<ffffffff8126b325>] xfs_map_blocks+0x155/0x230
[ 480.659711] [<ffffffff8126b597>] xfs_vm_writepage+0x197/0x540
[ 480.659713] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[ 480.659717] [<ffffffff810cc4a2>] __writepage+0x12/0x40
[ 480.659720] [<ffffffff810cd5f6>] write_cache_pages+0x236/0x4c0
[ 480.659722] [<ffffffff812c27ef>] ? xfs_log_commit_cil+0x3bf/0x480
[ 480.659724] [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
[ 480.659726] [<ffffffff812bccac>] ? xfs_trans_free+0x5c/0x70
[ 480.659728] [<ffffffff812bde3b>] ? xfs_trans_commit+0xab/0x260
[ 480.659731] [<ffffffff8127362e>] ? xfs_iunlock+0x7e/0xd0
[ 480.659733] [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
[ 480.659735] [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
[ 480.659737] [<ffffffff810cd90c>] do_writepages+0x1c/0x40
[ 480.659739] [<ffffffff810c3b7b>] __filemap_fdatawrite_range+0x5b/0x60
[ 480.659742] [<ffffffff810c3e1e>] filemap_fdatawrite_range+0xe/0x10
[ 480.659745] [<ffffffff811361ae>] sys_sync_file_range+0x14e/0x180
[ 480.659747] [<ffffffff81109240>] ? filp_close+0x60/0x90
[ 480.659749] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
[ 480.659756] INFO: task flush-8:64:3184 blocked for more than 120
seconds.
[ 480.659757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 480.659758] flush-8:64 D ffffffff8180e9c0 0 3184 2
0x00000000
[ 480.659761] ffff880dbcf31920 0000000000000046 ffff880dbdbcc830
0000000000012200
[ 480.659763] ffff880dbcf31fd8 ffff880dbcf30010 0000000000012200
0000000000012200
[ 480.659766] ffff880dbcf31fd8 0000000000012200 ffff881038e89810
ffff880dbdbcc830
[ 480.659768] Call Trace:
[ 480.659772] [<ffffffff810c34e0>] ? __lock_page+0x70/0x70
[ 480.659774] [<ffffffff816296e4>] schedule+0x24/0x70
[ 480.659777] [<ffffffff816297b7>] io_schedule+0x87/0xd0
[ 480.659779] [<ffffffff810c34e9>] sleep_on_page+0x9/0x10
[ 480.659781] [<ffffffff81627bd2>] __wait_on_bit_lock+0x52/0xb0
[ 480.659783] [<ffffffff810c547d>] ? find_get_pages_tag+0xcd/0x180
[ 480.659785] [<ffffffff810c34d2>] __lock_page+0x62/0x70
[ 480.659789] [<ffffffff8105de10>] ? autoremove_wake_function+0x40/0x40
[ 480.659791] [<ffffffff810cf190>] ? pagevec_lookup_tag+0x20/0x30
[ 480.659793] [<ffffffff810cd736>] write_cache_pages+0x376/0x4c0
[ 480.659796] [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
[ 480.659798] [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
[ 480.659800] [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
[ 480.659802] [<ffffffff810cd90c>] do_writepages+0x1c/0x40
[ 480.659805] [<ffffffff811308ff>] writeback_single_inode+0x18f/0x400
[ 480.659807] [<ffffffff81130d06>] writeback_sb_inodes+0x196/0x260
[ 480.659809] [<ffffffff81130e66>] __writeback_inodes_wb+0x96/0xc0
[ 480.659811] [<ffffffff811317bb>] wb_writeback+0x1eb/0x2c0
[ 480.659814] [<ffffffff8104c1a1>] ? try_to_del_timer_sync+0x81/0xe0
[ 480.659816] [<ffffffff81131a89>] wb_do_writeback+0x1f9/0x250
[ 480.659818] [<ffffffff81131b7a>] bdi_writeback_thread+0x9a/0x230
[ 480.659820] [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
[ 480.659822] [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
[ 480.659824] [<ffffffff8105d8d6>] kthread+0x96/0xa0
[ 480.659827] [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
[ 480.659829] [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
[ 480.659832] [<ffffffff8162c090>] ? gs_change+0xb/0xb
Am 26.06.2012 19:49, schrieb Stefan Priebe:
I would but both XFS and btrfs are crashing after a short period.
XFS crashes with this one:
[ 479.732636] INFO: task ceph-osd:3217 blocked for more than 120
seconds.
[ 479.747724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 479.763534] ceph-osd D ffffffff8180e9c0 0 3217 1
0x00000000
[ 479.779837] ffff880bc4321bd8 0000000000000082 ffff880bc5694830
0000000000012200
[ 479.779840] ffff880bc4321fd8 ffff880bc4320010 0000000000012200
0000000000012200
[ 479.779841] ffff880bc4321fd8 0000000000012200 ffff880e40ea9810
ffff880bc5694830
[ 479.779843] Call Trace:
[ 479.779850] [<ffffffff816296e4>] schedule+0x24/0x70
[ 479.779853] [<ffffffff812c2049>] xlog_wait+0x69/0x90
[ 479.779856] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[ 479.779858] [<ffffffff812c23b3>] xlog_cil_push+0x343/0x3c0
[ 479.779861] [<ffffffff8126ce09>] ? xfs_buf_unlock+0x19/0x70
[ 479.779862] [<ffffffff812c2ab1>] xlog_cil_force_lsn+0x101/0x110
[ 479.779864] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[ 479.779865] [<ffffffff812bcd77>] ? xfs_trans_free_items+0x87/0xb0
[ 479.779867] [<ffffffff812c07c8>] _xfs_log_force_lsn+0x48/0x290
[ 479.779871] [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
[ 479.779872] [<ffffffff812bdfdb>] xfs_trans_commit+0x24b/0x260
[ 479.779875] [<ffffffff81271e9d>] xfs_fs_log_dummy+0x5d/0x90
[ 479.779877] [<ffffffff812bed9c>] ? xfs_log_need_covered+0x7c/0xc0
[ 479.779879] [<ffffffff8127d378>] xfs_quiesce_data+0x88/0x90
[ 479.779881] [<ffffffff8127b428>] xfs_fs_sync_fs+0x28/0x60
[ 479.779884] [<ffffffff811363ae>] __sync_filesystem+0x5e/0x90
[ 479.779885] [<ffffffff811364b3>] sync_filesystem+0x43/0x60
[ 479.779887] [<ffffffff81136518>] sys_syncfs+0x48/0x80
[ 479.779890] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
Am 26.06.2012 18:59, schrieb Mark Nelson:
On 06/26/2012 11:43 AM, Stefan Priebe wrote:
Am 26.06.2012 18:29, schrieb Mark Nelson:
On 06/26/2012 11:15 AM, Stefan Priebe wrote:
Hi Stefan,
If you can, it would be really interesting to see the blktrace results
during these tests for both xfs and btrfs. blktrace is in the ubuntu
repositories and can be run quite easily from the command line during
your test.
Sure any special parameters? I have 4 SSDs per OSD Server.
Or just blktrace -o file?
Stefan
For each device you run it on you'll get one file per core. There may
be some performance impact if you run blktrace on every device per node.
If your data is well distributed, even a trace for one OSD (per test)
would be interesting.
so blktrace -o <outfile prefix> -d <device> where device is your first
OSD or something. If you can do it for both btrfs and xfs and maybe run
each test for a couple of minutes that might be enough.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
