On Mon, 15 Apr 2019 18:51:20 +0500 Roman Mamedov <rm@xxxxxxxxxxx> wrote: > At this point it starts rewriting data on both member devices into RAID5: > > [93729.643620] md/raid:md0: device sdg3 operational as raid disk 1 > [93729.643622] md/raid:md0: device sdi3 operational as raid disk 0 > [93729.674327] md/raid:md0: raid level 5 active with 2 out of 3 devices, algorithm 5 > [93729.750306] md: reshape of RAID array md0 > > md0 : active raid5 sdg3[1] sdi3[0] > 726982656 blocks super 1.2 level 5, 1024k chunk, algorithm 5 [3/2] [UU_] > [>....................] reshape = 1.5% (5624832/363491328) > finish=51.1min speed=116584K/sec > > Not what I was expecting. And this has some bugs too. I tried stopping this reshape as described on https://serverfault.com/questions/216508/how-to-interrupt-software-raid-resync after "echo frozen > /sys/block/md0/md/sync_action", the echo didn't even return, and all access to array immediately locked up, with md0_raid5 consuming 100% CPU. The entire system got due to no more being able to read from the array. I had to reboot with SysRQ + B. So much for the hope of a clean on-the-fly migration process for replacing a disk in RAID0. Sadly it seems mdadm quality is no longer what it used to be. [94639.822228] INFO: task btrfs-transacti:1667 blocked for more than 300 seconds. [94639.829533] Not tainted 4.14.111-rm1+ #21 [94639.836812] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [94639.844106] btrfs-transacti D 0 1667 2 0x80000000 [94639.844109] Call Trace: [94639.844117] ? __schedule+0x282/0x860 [94639.844120] ? finish_wait+0xb0/0xb0 [94639.844122] schedule+0x28/0x80 [94639.844130] md_write_start+0x165/0x240 [md_mod] [94639.844133] ? finish_wait+0xb0/0xb0 [94639.844135] ? finish_wait+0xb0/0xb0 [94639.844140] raid5_make_request+0x83/0xb20 [raid456] [94639.844147] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94639.844149] ? finish_wait+0xb0/0xb0 [94639.844150] ? finish_wait+0xb0/0xb0 [94639.844155] md_handle_request+0x119/0x190 [md_mod] [94639.844160] md_make_request+0x65/0x140 [md_mod] [94639.844162] generic_make_request+0xd1/0x310 [94639.844165] ? bio_alloc_bioset+0xbe/0x250 [94639.844166] submit_bio+0x4d/0x150 [94639.844168] ? bio_clone_fast+0x3a/0x80 [94639.844208] btrfs_map_bio+0x1f3/0x350 [btrfs] [94639.844226] btree_submit_bio_hook+0x8a/0xd0 [btrfs] [94639.844245] ? btrfs_map_block+0x15/0x20 [btrfs] [94639.844263] submit_one_bio+0x5f/0x80 [btrfs] [94639.844282] submit_extent_page+0x149/0x280 [btrfs] [94639.844301] ? end_extent_buffer_writeback+0x20/0x20 [btrfs] [94639.844319] write_one_eb+0x1a2/0x2c0 [btrfs] [94639.844337] ? end_extent_buffer_writeback+0x20/0x20 [btrfs] [94639.844356] btree_write_cache_pages+0x1d4/0x3a0 [btrfs] [94639.844360] do_writepages+0x41/0xd0 [94639.844362] ? __set_page_dirty_nobuffers+0x13e/0x160 [94639.844379] ? merge_state.part.44+0x4d/0x160 [btrfs] [94639.844397] ? clear_state_bit+0xda/0x180 [btrfs] [94639.844416] ? alloc_extent_state+0x6c/0xe0 [btrfs] [94639.844418] __filemap_fdatawrite_range+0xc1/0x100 [94639.844436] btrfs_write_marked_extents+0x59/0x130 [btrfs] [94639.844453] btrfs_write_and_wait_marked_extents.constprop.18+0x4f/0x90 [btrfs] [94639.844470] btrfs_commit_transaction+0x551/0x920 [btrfs] [94639.844488] ? start_transaction+0x9d/0x430 [btrfs] [94639.844505] transaction_kthread+0x187/0x1b0 [btrfs] [94639.844522] ? btrfs_cleanup_transaction+0x590/0x590 [btrfs] [94639.844525] kthread+0x11a/0x130 [94639.844527] ? __kthread_parkme+0xa0/0xa0 [94639.844528] ret_from_fork+0x22/0x40 [94639.844563] INFO: task kvm-android:14108 blocked for more than 300 seconds. [94639.851785] Not tainted 4.14.111-rm1+ #21 [94639.858941] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [94639.866186] kvm-android D 0 14108 1 0x00000000 [94639.866192] Call Trace: [94639.866197] ? __schedule+0x282/0x860 [94639.866200] ? finish_wait+0xb0/0xb0 [94639.866206] schedule+0x28/0x80 [94639.866218] md_write_start+0x165/0x240 [md_mod] [94639.866230] ? finish_wait+0xb0/0xb0 [94639.866236] ? finish_wait+0xb0/0xb0 [94639.866246] raid5_make_request+0x83/0xb20 [raid456] [94639.866256] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94639.866257] ? finish_wait+0xb0/0xb0 [94639.866259] ? finish_wait+0xb0/0xb0 [94639.866263] md_handle_request+0x119/0x190 [md_mod] [94639.866268] md_make_request+0x65/0x140 [md_mod] [94639.866270] generic_make_request+0xd1/0x310 [94639.866275] submit_bio+0x4d/0x150 [94639.866301] btrfs_map_bio+0x1f3/0x350 [btrfs] [94639.866319] btrfs_submit_bio_hook+0x97/0x190 [btrfs] [94639.866340] submit_one_bio+0x5f/0x80 [btrfs] [94639.866365] submit_extent_page+0x149/0x280 [btrfs] [94639.866394] ? end_extent_writepage+0x90/0x90 [btrfs] [94639.866423] __extent_writepage_io+0x292/0x4d0 [btrfs] [94639.866452] ? end_extent_writepage+0x90/0x90 [btrfs] [94639.866481] __extent_writepage+0x135/0x300 [btrfs] [94639.866506] extent_write_cache_pages.constprop.53+0x15f/0x3f0 [btrfs] [94639.866525] extent_writepages+0x60/0xa0 [btrfs] [94639.866528] ? hrtimer_try_to_cancel+0x78/0x100 [94639.866545] ? btrfs_set_bit_hook+0x2f0/0x2f0 [btrfs] [94639.866547] do_writepages+0x41/0xd0 [94639.866549] ? futex_wait+0x1cb/0x240 [94639.866551] ? __hrtimer_init+0x90/0x90 [94639.866553] __filemap_fdatawrite_range+0xc1/0x100 [94639.866571] btrfs_fdatawrite_range+0x29/0x70 [btrfs] [94639.866589] start_ordered_ops+0x50/0x90 [btrfs] [94639.866610] btrfs_sync_file+0x9b/0x480 [btrfs] [94639.866614] ? selinux_file_permission+0x7b/0x140 [94639.866619] do_fsync+0x43/0x80 [94639.866621] SyS_fdatasync+0xf/0x20 [94639.866623] do_syscall_64+0x6e/0x100 [94639.866625] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94639.866627] RIP: 0033:0x7f6bb4a5184d [94639.866628] RSP: 002b:00007f6b092d0970 EFLAGS: 00000293 ORIG_RAX: 000000000000004b [94639.866630] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f6bb4a5184d [94639.866631] RDX: 000055b9aacc2ce0 RSI: 000055b9a80fb658 RDI: 0000000000000011 [94639.866632] RBP: 00007f6ba4689a10 R08: 0000000000000000 R09: 00000000ffffffff [94639.866633] R10: 00007f6b092d09a0 R11: 0000000000000293 R12: 00007f6ba46875b0 [94639.866634] R13: 000055b9aacc2d38 R14: 00007f6b08ad1000 R15: 0000000000000003 [94639.866649] INFO: task kvm-tsuki:14113 blocked for more than 300 seconds. [94639.873891] Not tainted 4.14.111-rm1+ #21 [94639.881077] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [94639.888318] kvm-tsuki D 0 14113 1 0x00000000 [94639.888321] Call Trace: [94639.888326] ? __schedule+0x282/0x860 [94639.888328] ? finish_wait+0xb0/0xb0 [94639.888330] schedule+0x28/0x80 [94639.888337] md_write_start+0x165/0x240 [md_mod] [94639.888339] ? finish_wait+0xb0/0xb0 [94639.888341] ? finish_wait+0xb0/0xb0 [94639.888346] raid5_make_request+0x83/0xb20 [raid456] [94639.888358] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94639.888359] ? finish_wait+0xb0/0xb0 [94639.888362] ? finish_wait+0xb0/0xb0 [94639.888367] md_handle_request+0x119/0x190 [md_mod] [94639.888381] md_make_request+0x65/0x140 [md_mod] [94639.888388] generic_make_request+0xd1/0x310 [94639.888396] submit_bio+0x4d/0x150 [94639.888427] btrfs_map_bio+0x1f3/0x350 [btrfs] [94639.888452] btrfs_submit_bio_hook+0x97/0x190 [btrfs] [94639.888480] submit_one_bio+0x5f/0x80 [btrfs] [94639.888502] submit_extent_page+0x149/0x280 [btrfs] [94639.888530] ? end_extent_writepage+0x90/0x90 [btrfs] [94639.888559] __extent_writepage_io+0x292/0x4d0 [btrfs] [94639.888585] ? end_extent_writepage+0x90/0x90 [btrfs] [94639.888604] __extent_writepage+0x135/0x300 [btrfs] [94639.888626] extent_write_cache_pages.constprop.53+0x15f/0x3f0 [btrfs] [94639.888628] ? __schedule+0x28a/0x860 [94639.888649] extent_writepages+0x60/0xa0 [btrfs] [94639.888652] ? hrtimer_try_to_cancel+0x78/0x100 [94639.888669] ? btrfs_set_bit_hook+0x2f0/0x2f0 [btrfs] [94639.888673] do_writepages+0x41/0xd0 [94639.888675] ? futex_wait+0x1cb/0x240 [94639.888677] ? __hrtimer_init+0x90/0x90 [94639.888680] __filemap_fdatawrite_range+0xc1/0x100 [94639.888698] btrfs_fdatawrite_range+0x29/0x70 [btrfs] [94639.888718] start_ordered_ops+0x50/0x90 [btrfs] [94639.888736] btrfs_sync_file+0x9b/0x480 [btrfs] [94639.888742] do_fsync+0x43/0x80 [94639.888744] SyS_fdatasync+0xf/0x20 [94639.888746] do_syscall_64+0x6e/0x100 [94639.888748] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94639.888750] RIP: 0033:0x7faed28a984d [94639.888751] RSP: 002b:00007fae2dcc0970 EFLAGS: 00000293 ORIG_RAX: 000000000000004b [94639.888752] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007faed28a984d [94639.888753] RDX: 0000561184a5ac30 RSI: 00005611821c9658 RDI: 0000000000000011 [94639.888754] RBP: 0000561184d997c0 R08: 0000000000000000 R09: 00000000ffffffff [94639.888755] R10: 00007fae2dcc09a0 R11: 0000000000000293 R12: 0000561185567280 [94639.888756] R13: 0000561184a5ac88 R14: 00007fae2d4c1000 R15: 0000000000000003 [94639.888761] INFO: task kworker/u16:0:13692 blocked for more than 300 seconds. [94639.895950] Not tainted 4.14.111-rm1+ #21 [94639.903136] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [94639.910362] kworker/u16:0 D 0 13692 2 0x80000000 [94639.910367] Workqueue: writeback wb_workfn (flush-btrfs-1) [94639.910369] Call Trace: [94639.910374] ? __schedule+0x282/0x860 [94639.910376] ? finish_wait+0xb0/0xb0 [94639.910378] schedule+0x28/0x80 [94639.910384] md_write_start+0x165/0x240 [md_mod] [94639.910386] ? finish_wait+0xb0/0xb0 [94639.910388] ? finish_wait+0xb0/0xb0 [94639.910392] raid5_make_request+0x83/0xb20 [raid456] [94639.910399] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94639.910400] ? finish_wait+0xb0/0xb0 [94639.910402] ? finish_wait+0xb0/0xb0 [94639.910406] md_handle_request+0x119/0x190 [md_mod] [94639.910412] md_make_request+0x65/0x140 [md_mod] [94639.910414] generic_make_request+0xd1/0x310 [94639.910416] submit_bio+0x4d/0x150 [94639.910441] btrfs_map_bio+0x1f3/0x350 [btrfs] [94639.910458] btrfs_submit_bio_hook+0x97/0x190 [btrfs] [94639.910477] submit_one_bio+0x5f/0x80 [btrfs] [94639.910495] flush_epd_write_bio.isra.39+0x1e/0x30 [btrfs] [94639.910513] extent_writepages+0x73/0xa0 [btrfs] [94639.910531] ? btrfs_set_bit_hook+0x2f0/0x2f0 [btrfs] [94639.910533] do_writepages+0x41/0xd0 [94639.910535] ? __wb_calc_thresh+0x3a/0x130 [94639.910536] ? wb_calc_thresh+0x4f/0x70 [94639.910538] __writeback_single_inode+0x3d/0x360 [94639.910540] writeback_sb_inodes+0x234/0x510 [94639.910542] __writeback_inodes_wb+0x5d/0xb0 [94639.910544] wb_writeback+0x262/0x2f0 [94639.910546] wb_workfn+0x28d/0x380 [94639.910548] ? __switch_to+0x1ee/0x430 [94639.910551] process_one_work+0x18a/0x3a0 [94639.910553] ? create_worker+0x1c0/0x1c0 [94639.910555] worker_thread+0x2e/0x3d0 [94639.910556] ? create_worker+0x1c0/0x1c0 [94639.910558] kthread+0x11a/0x130 [94639.910560] ? __kthread_parkme+0xa0/0xa0 [94639.910561] ret_from_fork+0x22/0x40 [94639.910565] INFO: task bash:13876 blocked for more than 300 seconds. [94639.917755] Not tainted 4.14.111-rm1+ #21 [94639.924865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [94639.932039] bash D 0 13876 13870 0x00000000 [94639.932042] Call Trace: [94639.932046] ? __schedule+0x282/0x860 [94639.932049] ? account_entity_enqueue+0xf2/0x120 [94639.932051] schedule+0x28/0x80 [94639.932053] schedule_timeout+0x292/0x370 [94639.932054] ? enqueue_task_fair+0x5a/0x520 [94639.932056] ? select_idle_sibling+0x23/0x360 [94639.932057] wait_for_completion+0x140/0x1b0 [94639.932059] ? wake_up_q+0x80/0x80 [94639.932063] kthread_stop+0x42/0x100 [94639.932070] md_unregister_thread+0x4c/0x90 [md_mod] [94639.932083] md_reap_sync_thread+0x15/0x170 [md_mod] [94639.932088] action_store+0x164/0x2c0 [md_mod] [94639.932104] md_attr_store+0x91/0xe0 [md_mod] [94639.932112] kernfs_fop_write+0x150/0x1d0 [94639.932120] __vfs_write+0x36/0x170 [94639.932128] ? selinux_file_permission+0x117/0x140 [94639.932134] ? security_file_permission+0x3a/0xb0 [94639.932140] vfs_write+0xb9/0x1c0 [94639.932146] SyS_write+0x64/0xe0 [94639.932152] do_syscall_64+0x6e/0x100 [94639.932170] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94639.932177] RIP: 0033:0x7fee83903970 [94639.932182] RSP: 002b:00007ffd41b1a828 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [94639.932184] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007fee83903970 [94639.932185] RDX: 0000000000000007 RSI: 00000000026d1008 RDI: 0000000000000001 [94639.932186] RBP: 00000000026d1008 R08: 00007fee83bc3760 R09: 00007fee84227700 [94639.932187] R10: 0000000000000073 R11: 0000000000000246 R12: 0000000000000007 [94639.932188] R13: 0000000000000001 R14: 00007fee83bc2600 R15: 0000000000000007 [94639.932358] INFO: task md0_resync:14081 blocked for more than 300 seconds. [94639.939477] Not tainted 4.14.111-rm1+ #21 [94639.946514] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [94639.953621] md0_resync D 0 14081 2 0x80000000 [94639.953623] Call Trace: [94639.953631] ? __schedule+0x282/0x860 [94639.953634] schedule+0x28/0x80 [94639.953644] md_do_sync.cold.80+0x7f1/0x8be [md_mod] [94639.953647] ? finish_wait+0xb0/0xb0 [94639.953649] ? __switch_to_asm+0x40/0x70 [94639.953654] ? find_pers+0x70/0x70 [md_mod] [94639.953658] md_thread+0x9d/0x160 [md_mod] [94639.953661] kthread+0x11a/0x130 [94639.953663] ? __kthread_parkme+0xa0/0xa0 [94639.953665] ret_from_fork+0x22/0x40 [94664.072381] sysrq: SysRq : Show Blocked State [94664.079569] task PC stack pid father [94664.079662] btrfs-transacti D 0 1667 2 0x80000000 [94664.079668] Call Trace: [94664.079682] ? __schedule+0x282/0x860 [94664.079688] ? finish_wait+0xb0/0xb0 [94664.079692] schedule+0x28/0x80 [94664.079710] md_write_start+0x165/0x240 [md_mod] [94664.079715] ? finish_wait+0xb0/0xb0 [94664.079718] ? finish_wait+0xb0/0xb0 [94664.079729] raid5_make_request+0x83/0xb20 [raid456] [94664.079744] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94664.079747] ? finish_wait+0xb0/0xb0 [94664.079751] ? finish_wait+0xb0/0xb0 [94664.079762] md_handle_request+0x119/0x190 [md_mod] [94664.079775] md_make_request+0x65/0x140 [md_mod] [94664.079781] generic_make_request+0xd1/0x310 [94664.079786] ? bio_alloc_bioset+0xbe/0x250 [94664.079789] submit_bio+0x4d/0x150 [94664.079792] ? bio_clone_fast+0x3a/0x80 [94664.079851] btrfs_map_bio+0x1f3/0x350 [btrfs] [94664.079898] btree_submit_bio_hook+0x8a/0xd0 [btrfs] [94664.079949] ? btrfs_map_block+0x15/0x20 [btrfs] [94664.079998] submit_one_bio+0x5f/0x80 [btrfs] [94664.080048] submit_extent_page+0x149/0x280 [btrfs] [94664.080099] ? end_extent_buffer_writeback+0x20/0x20 [btrfs] [94664.080149] write_one_eb+0x1a2/0x2c0 [btrfs] [94664.080200] ? end_extent_buffer_writeback+0x20/0x20 [btrfs] [94664.080250] btree_write_cache_pages+0x1d4/0x3a0 [btrfs] [94664.080257] do_writepages+0x41/0xd0 [94664.080261] ? __set_page_dirty_nobuffers+0x13e/0x160 [94664.080311] ? merge_state.part.44+0x4d/0x160 [btrfs] [94664.080359] ? clear_state_bit+0xda/0x180 [btrfs] [94664.080408] ? alloc_extent_state+0x6c/0xe0 [btrfs] [94664.080413] __filemap_fdatawrite_range+0xc1/0x100 [94664.080460] btrfs_write_marked_extents+0x59/0x130 [btrfs] [94664.080506] btrfs_write_and_wait_marked_extents.constprop.18+0x4f/0x90 [btrfs] [94664.080553] btrfs_commit_transaction+0x551/0x920 [btrfs] [94664.080600] ? start_transaction+0x9d/0x430 [btrfs] [94664.080645] transaction_kthread+0x187/0x1b0 [btrfs] [94664.080692] ? btrfs_cleanup_transaction+0x590/0x590 [btrfs] [94664.080696] kthread+0x11a/0x130 [94664.080700] ? __kthread_parkme+0xa0/0xa0 [94664.080704] ret_from_fork+0x22/0x40 [94664.080838] qbittorrent-nox D 0 3588 1 0x00000000 [94664.080842] Call Trace: [94664.080848] ? __schedule+0x282/0x860 [94664.080853] schedule+0x28/0x80 [94664.080858] io_schedule+0x12/0x40 [94664.080862] generic_file_read_iter+0x3ce/0xa10 [94664.080867] ? page_cache_tree_insert+0xd0/0xd0 [94664.080872] do_iter_readv_writev+0x120/0x180 [94664.080876] do_iter_read+0xfd/0x1c0 [94664.080879] vfs_readv+0x7a/0xc0 [94664.080885] ? SyS_epoll_ctl+0x11b/0xe30 [94664.080890] ? SyS_futex+0x152/0x1a2 [94664.080893] do_preadv+0xaa/0x100 [94664.080898] do_syscall_64+0x6e/0x100 [94664.080902] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.080906] RIP: 0033:0x7fca4ad41133 [94664.080908] RSP: 002b:00007fca37ffe4c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000127 [94664.080912] RAX: ffffffffffffffda RBX: 00007fca37ffe5c0 RCX: 00007fca4ad41133 [94664.080914] RDX: 0000000000000020 RSI: 00007fca37ffe5c0 RDI: 0000000000000054 [94664.080916] RBP: 0000000000000000 R08: 0000000000000020 R09: 00007fca37ffe560 [94664.080918] R10: 0000000046034000 R11: 0000000000000293 R12: 0000000000000020 [94664.080919] R13: 00007fca4ad41100 R14: 0000000000000020 R15: 0000000046034000 [94664.080923] qbittorrent-nox D 0 3589 1 0x00000000 [94664.080927] Call Trace: [94664.080932] ? __schedule+0x282/0x860 [94664.080937] schedule+0x28/0x80 [94664.080941] io_schedule+0x12/0x40 [94664.080945] generic_file_read_iter+0x3ce/0xa10 [94664.080949] ? page_cache_tree_insert+0xd0/0xd0 [94664.080953] do_iter_readv_writev+0x120/0x180 [94664.080956] do_iter_read+0xfd/0x1c0 [94664.080960] vfs_readv+0x7a/0xc0 [94664.080964] ? SyS_epoll_ctl+0x11b/0xe30 [94664.080969] ? SyS_futex+0x152/0x1a2 [94664.080972] do_preadv+0xaa/0x100 [94664.080976] do_syscall_64+0x6e/0x100 [94664.080980] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.080983] RIP: 0033:0x7fca4ad41133 [94664.080984] RSP: 002b:00007fca377fd4c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000127 [94664.080987] RAX: ffffffffffffffda RBX: 00007fca377fd5c0 RCX: 00007fca4ad41133 [94664.080989] RDX: 0000000000000020 RSI: 00007fca377fd5c0 RDI: 0000000000000054 [94664.080991] RBP: 0000000000000000 R08: 0000000000000020 R09: 00007fca377fd560 [94664.080993] R10: 000000001810c000 R11: 0000000000000293 R12: 0000000000000020 [94664.080994] R13: 00007fca4ad41100 R14: 0000000000000020 R15: 000000001810c000 [94664.080998] qbittorrent-nox D 0 3590 1 0x00000000 [94664.081001] Call Trace: [94664.081006] ? __schedule+0x282/0x860 [94664.081011] schedule+0x28/0x80 [94664.081015] io_schedule+0x12/0x40 [94664.081019] generic_file_read_iter+0x3ce/0xa10 [94664.081023] ? page_cache_tree_insert+0xd0/0xd0 [94664.081027] do_iter_readv_writev+0x120/0x180 [94664.081030] do_iter_read+0xfd/0x1c0 [94664.081034] vfs_readv+0x7a/0xc0 [94664.081038] ? SyS_epoll_ctl+0x11b/0xe30 [94664.081042] ? SyS_futex+0x152/0x1a2 [94664.081046] do_preadv+0xaa/0x100 [94664.081050] do_syscall_64+0x6e/0x100 [94664.081054] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.081056] RIP: 0033:0x7fca4ad41133 [94664.081058] RSP: 002b:00007fca36ffc4c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000127 [94664.081061] RAX: ffffffffffffffda RBX: 00007fca36ffc5c0 RCX: 00007fca4ad41133 [94664.081063] RDX: 0000000000000020 RSI: 00007fca36ffc5c0 RDI: 0000000000000054 [94664.081065] RBP: 0000000000000000 R08: 0000000000000020 R09: 00007fca36ffc560 [94664.081066] R10: 000000001dc00000 R11: 0000000000000293 R12: 0000000000000020 [94664.081068] R13: 00007fca4ad41100 R14: 0000000000000020 R15: 000000001dc00000 [94664.081084] kvm-android D 0 14108 1 0x00000000 [94664.081087] Call Trace: [94664.081093] ? __schedule+0x282/0x860 [94664.081096] ? finish_wait+0xb0/0xb0 [94664.081100] schedule+0x28/0x80 [94664.081113] md_write_start+0x165/0x240 [md_mod] [94664.081117] ? finish_wait+0xb0/0xb0 [94664.081120] ? finish_wait+0xb0/0xb0 [94664.081130] raid5_make_request+0x83/0xb20 [raid456] [94664.081143] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94664.081146] ? finish_wait+0xb0/0xb0 [94664.081150] ? finish_wait+0xb0/0xb0 [94664.081161] md_handle_request+0x119/0x190 [md_mod] [94664.081174] md_make_request+0x65/0x140 [md_mod] [94664.081178] generic_make_request+0xd1/0x310 [94664.081181] submit_bio+0x4d/0x150 [94664.081232] btrfs_map_bio+0x1f3/0x350 [btrfs] [94664.081280] btrfs_submit_bio_hook+0x97/0x190 [btrfs] [94664.081330] submit_one_bio+0x5f/0x80 [btrfs] [94664.081380] submit_extent_page+0x149/0x280 [btrfs] [94664.081430] ? end_extent_writepage+0x90/0x90 [btrfs] [94664.081480] __extent_writepage_io+0x292/0x4d0 [btrfs] [94664.081530] ? end_extent_writepage+0x90/0x90 [btrfs] [94664.081581] __extent_writepage+0x135/0x300 [btrfs] [94664.081632] extent_write_cache_pages.constprop.53+0x15f/0x3f0 [btrfs] [94664.081684] extent_writepages+0x60/0xa0 [btrfs] [94664.081690] ? hrtimer_try_to_cancel+0x78/0x100 [94664.081737] ? btrfs_set_bit_hook+0x2f0/0x2f0 [btrfs] [94664.081740] do_writepages+0x41/0xd0 [94664.081744] ? futex_wait+0x1cb/0x240 [94664.081749] ? __hrtimer_init+0x90/0x90 [94664.081757] __filemap_fdatawrite_range+0xc1/0x100 [94664.081806] btrfs_fdatawrite_range+0x29/0x70 [btrfs] [94664.081854] start_ordered_ops+0x50/0x90 [btrfs] [94664.081904] btrfs_sync_file+0x9b/0x480 [btrfs] [94664.081909] ? selinux_file_permission+0x7b/0x140 [94664.081915] do_fsync+0x43/0x80 [94664.081919] SyS_fdatasync+0xf/0x20 [94664.081923] do_syscall_64+0x6e/0x100 [94664.081927] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.081930] RIP: 0033:0x7f6bb4a5184d [94664.081931] RSP: 002b:00007f6b092d0970 EFLAGS: 00000293 ORIG_RAX: 000000000000004b [94664.081934] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f6bb4a5184d [94664.081936] RDX: 000055b9aacc2ce0 RSI: 000055b9a80fb658 RDI: 0000000000000011 [94664.081938] RBP: 00007f6ba4689a10 R08: 0000000000000000 R09: 00000000ffffffff [94664.081940] R10: 00007f6b092d09a0 R11: 0000000000000293 R12: 00007f6ba46875b0 [94664.081941] R13: 000055b9aacc2d38 R14: 00007f6b08ad1000 R15: 0000000000000003 [94664.081976] qbittorrent-nox D 0 2755 1 0x00000000 [94664.081979] Call Trace: [94664.081985] ? __schedule+0x282/0x860 [94664.081989] schedule+0x28/0x80 [94664.081993] io_schedule+0x12/0x40 [94664.081998] generic_file_read_iter+0x3ce/0xa10 [94664.082002] ? page_cache_tree_insert+0xd0/0xd0 [94664.082006] do_iter_readv_writev+0x120/0x180 [94664.082010] do_iter_read+0xfd/0x1c0 [94664.082013] vfs_readv+0x7a/0xc0 [94664.082017] ? SyS_epoll_ctl+0x11b/0xe30 [94664.082021] ? lockref_put_or_lock+0x59/0x80 [94664.082025] ? dput.part.34+0x165/0x200 [94664.082029] ? SyS_futex+0x152/0x1a2 [94664.082032] do_preadv+0xaa/0x100 [94664.082036] do_syscall_64+0x6e/0x100 [94664.082040] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.082043] RIP: 0033:0x7f2dc32e1133 [94664.082044] RSP: 002b:00007f2dbc8dd4c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000127 [94664.082047] RAX: ffffffffffffffda RBX: 00007f2dbc8dd5c0 RCX: 00007f2dc32e1133 [94664.082049] RDX: 0000000000000020 RSI: 00007f2dbc8dd5c0 RDI: 000000000000004e [94664.082051] RBP: 0000000000000000 R08: 0000000000000020 R09: 00007f2dbc8dd560 [94664.082052] R10: 00000000132983fc R11: 0000000000000293 R12: 0000000000000020 [94664.082054] R13: 00007f2dc32e1100 R14: 0000000000000020 R15: 00000000132983fc [94664.082067] kvm-tsuki D 0 14113 1 0x00000000 [94664.082070] Call Trace: [94664.082076] ? __schedule+0x282/0x860 [94664.082079] ? finish_wait+0xb0/0xb0 [94664.082083] schedule+0x28/0x80 [94664.082096] md_write_start+0x165/0x240 [md_mod] [94664.082100] ? finish_wait+0xb0/0xb0 [94664.082103] ? finish_wait+0xb0/0xb0 [94664.082112] raid5_make_request+0x83/0xb20 [raid456] [94664.082124] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94664.082128] ? finish_wait+0xb0/0xb0 [94664.082132] ? finish_wait+0xb0/0xb0 [94664.082143] md_handle_request+0x119/0x190 [md_mod] [94664.082156] md_make_request+0x65/0x140 [md_mod] [94664.082160] generic_make_request+0xd1/0x310 [94664.082163] submit_bio+0x4d/0x150 [94664.082217] btrfs_map_bio+0x1f3/0x350 [btrfs] [94664.082288] btrfs_submit_bio_hook+0x97/0x190 [btrfs] [94664.082341] submit_one_bio+0x5f/0x80 [btrfs] [94664.082393] submit_extent_page+0x149/0x280 [btrfs] [94664.082446] ? end_extent_writepage+0x90/0x90 [btrfs] [94664.082497] __extent_writepage_io+0x292/0x4d0 [btrfs] [94664.082550] ? end_extent_writepage+0x90/0x90 [btrfs] [94664.082603] __extent_writepage+0x135/0x300 [btrfs] [94664.082657] extent_write_cache_pages.constprop.53+0x15f/0x3f0 [btrfs] [94664.082665] ? __schedule+0x28a/0x860 [94664.082717] extent_writepages+0x60/0xa0 [btrfs] [94664.082725] ? hrtimer_try_to_cancel+0x78/0x100 [94664.082773] ? btrfs_set_bit_hook+0x2f0/0x2f0 [btrfs] [94664.082780] do_writepages+0x41/0xd0 [94664.082785] ? futex_wait+0x1cb/0x240 [94664.082793] ? __hrtimer_init+0x90/0x90 [94664.082800] __filemap_fdatawrite_range+0xc1/0x100 [94664.082851] btrfs_fdatawrite_range+0x29/0x70 [btrfs] [94664.082901] start_ordered_ops+0x50/0x90 [btrfs] [94664.082954] btrfs_sync_file+0x9b/0x480 [btrfs] [94664.082962] do_fsync+0x43/0x80 [94664.082969] SyS_fdatasync+0xf/0x20 [94664.082975] do_syscall_64+0x6e/0x100 [94664.082981] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.082986] RIP: 0033:0x7faed28a984d [94664.082991] RSP: 002b:00007fae2dcc0970 EFLAGS: 00000293 ORIG_RAX: 000000000000004b [94664.082995] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007faed28a984d [94664.082997] RDX: 0000561184a5ac30 RSI: 00005611821c9658 RDI: 0000000000000011 [94664.082999] RBP: 0000561184d997c0 R08: 0000000000000000 R09: 00000000ffffffff [94664.083001] R10: 00007fae2dcc09a0 R11: 0000000000000293 R12: 0000561185567280 [94664.083002] R13: 0000561184a5ac88 R14: 00007fae2d4c1000 R15: 0000000000000003 [94664.083012] kvm-tsuki D 0 14170 1 0x00000000 [94664.083022] Call Trace: [94664.083030] ? __schedule+0x282/0x860 [94664.083039] schedule+0x28/0x80 [94664.083047] io_schedule+0x12/0x40 [94664.083054] wait_on_page_bit+0x114/0x150 [94664.083061] ? page_cache_tree_insert+0xd0/0xd0 [94664.083114] prepare_pages+0x1eb/0x210 [btrfs] [94664.083167] __btrfs_buffered_write+0x2b5/0x750 [btrfs] [94664.083221] btrfs_file_write_iter+0x3dd/0x580 [btrfs] [94664.083229] __vfs_write+0x127/0x170 [94664.083234] vfs_write+0xb9/0x1c0 [94664.083238] SyS_pwrite64+0x9c/0xd0 [94664.083244] ? __switch_to+0x1ee/0x430 [94664.083248] do_syscall_64+0x6e/0x100 [94664.083253] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.083256] RIP: 0033:0x7faed2b78983 [94664.083258] RSP: 002b:00007fae2e4c1950 EFLAGS: 00000293 ORIG_RAX: 0000000000000012 [94664.083264] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007faed2b78983 [94664.083266] RDX: 0000000000000200 RSI: 00007faeb6f60400 RDI: 0000000000000011 [94664.083268] RBP: 0000561184dbaa60 R08: 0000000000000000 R09: 00007fae2e4c2700 [94664.083270] R10: 0000000016cf5000 R11: 0000000000000293 R12: 00007faeb6f60400 [94664.083274] R13: 0000561184a5ac88 R14: 00007fae2dcc2000 R15: 0000000000000003 [94664.083278] kvm-tsuki D 0 14174 1 0x00000000 [94664.083284] Call Trace: [94664.083290] ? __schedule+0x282/0x860 [94664.083298] schedule+0x28/0x80 [94664.083302] rwsem_down_write_failed+0x184/0x3b0 [94664.083310] call_rwsem_down_write_failed+0x13/0x20 [94664.083315] down_write+0x29/0x40 [94664.083371] btrfs_file_write_iter+0xba/0x580 [btrfs] [94664.083379] __vfs_write+0x127/0x170 [94664.083384] vfs_write+0xb9/0x1c0 [94664.083390] SyS_pwrite64+0x9c/0xd0 [94664.083395] ? __switch_to+0x1ee/0x430 [94664.083401] do_syscall_64+0x6e/0x100 [94664.083406] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.083412] RIP: 0033:0x7faed2b78983 [94664.083415] RSP: 002b:00007fae2ccbe950 EFLAGS: 00000293 ORIG_RAX: 0000000000000012 [94664.083419] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007faed2b78983 [94664.083422] RDX: 0000000000000400 RSI: 00007fae7f233000 RDI: 0000000000000011 [94664.083425] RBP: 0000561185534250 R08: 0000000000000000 R09: 00007fae2ccbf700 [94664.083427] R10: 00000000ccfbd000 R11: 0000000000000293 R12: 00007fae7f233000 [94664.083429] R13: 0000561184a5ac88 R14: 00007fae2c4bf000 R15: 0000000000000003 [94664.083435] kvm-tsuki D 0 14750 1 0x00000000 [94664.083441] Call Trace: [94664.083448] ? __schedule+0x282/0x860 [94664.083455] schedule+0x28/0x80 [94664.083459] rwsem_down_write_failed+0x184/0x3b0 [94664.083466] call_rwsem_down_write_failed+0x13/0x20 [94664.083471] down_write+0x29/0x40 [94664.083526] btrfs_file_write_iter+0xba/0x580 [btrfs] [94664.083534] __vfs_write+0x127/0x170 [94664.083539] vfs_write+0xb9/0x1c0 [94664.083544] SyS_pwrite64+0x9c/0xd0 [94664.083549] ? __switch_to+0x1ee/0x430 [94664.083555] do_syscall_64+0x6e/0x100 [94664.083560] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.083564] RIP: 0033:0x7faed2b78983 [94664.083567] RSP: 002b:00007fae2f4c3950 EFLAGS: 00000293 ORIG_RAX: 0000000000000012 [94664.083571] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007faed2b78983 [94664.083574] RDX: 0000000000001000 RSI: 00007faea85cb000 RDI: 0000000000000011 [94664.083577] RBP: 0000561185498c80 R08: 0000000000000000 R09: 00007fae2f4c4700 [94664.083580] R10: 000000001e42c000 R11: 0000000000000293 R12: 00007faea85cb000 [94664.083582] R13: 0000561184a5ac88 R14: 00007fae2ecc4000 R15: 0000000000000003 [94664.083600] kworker/u16:0 D 0 13692 2 0x80000000 [94664.083611] Workqueue: writeback wb_workfn (flush-btrfs-1) [94664.083616] Call Trace: [94664.083623] ? __schedule+0x282/0x860 [94664.083630] ? finish_wait+0xb0/0xb0 [94664.083635] schedule+0x28/0x80 [94664.083651] md_write_start+0x165/0x240 [md_mod] [94664.083657] ? finish_wait+0xb0/0xb0 [94664.083661] ? finish_wait+0xb0/0xb0 [94664.083672] raid5_make_request+0x83/0xb20 [raid456] [94664.083688] ? dm_dax_direct_access+0x140/0x140 [dm_mod] [94664.083695] ? finish_wait+0xb0/0xb0 [94664.083700] ? finish_wait+0xb0/0xb0 [94664.083714] md_handle_request+0x119/0x190 [md_mod] [94664.083731] md_make_request+0x65/0x140 [md_mod] [94664.083738] generic_make_request+0xd1/0x310 [94664.083743] submit_bio+0x4d/0x150 [94664.083802] btrfs_map_bio+0x1f3/0x350 [btrfs] [94664.083858] btrfs_submit_bio_hook+0x97/0x190 [btrfs] [94664.083917] submit_one_bio+0x5f/0x80 [btrfs] [94664.083975] flush_epd_write_bio.isra.39+0x1e/0x30 [btrfs] [94664.084034] extent_writepages+0x73/0xa0 [btrfs] [94664.084088] ? btrfs_set_bit_hook+0x2f0/0x2f0 [btrfs] [94664.084094] do_writepages+0x41/0xd0 [94664.084099] ? __wb_calc_thresh+0x3a/0x130 [94664.084104] ? wb_calc_thresh+0x4f/0x70 [94664.084109] __writeback_single_inode+0x3d/0x360 [94664.084114] writeback_sb_inodes+0x234/0x510 [94664.084121] __writeback_inodes_wb+0x5d/0xb0 [94664.084126] wb_writeback+0x262/0x2f0 [94664.084132] wb_workfn+0x28d/0x380 [94664.084137] ? __switch_to+0x1ee/0x430 [94664.084145] process_one_work+0x18a/0x3a0 [94664.084150] ? create_worker+0x1c0/0x1c0 [94664.084156] worker_thread+0x2e/0x3d0 [94664.084162] ? create_worker+0x1c0/0x1c0 [94664.084167] kthread+0x11a/0x130 [94664.084171] ? __kthread_parkme+0xa0/0xa0 [94664.084176] ret_from_fork+0x22/0x40 [94664.084188] bash D 0 13876 13870 0x00000000 [94664.084193] Call Trace: [94664.084201] ? __schedule+0x282/0x860 [94664.084206] ? account_entity_enqueue+0xf2/0x120 [94664.084213] schedule+0x28/0x80 [94664.084217] schedule_timeout+0x292/0x370 [94664.084223] ? enqueue_task_fair+0x5a/0x520 [94664.084227] ? select_idle_sibling+0x23/0x360 [94664.084232] wait_for_completion+0x140/0x1b0 [94664.084237] ? wake_up_q+0x80/0x80 [94664.084244] kthread_stop+0x42/0x100 [94664.084259] md_unregister_thread+0x4c/0x90 [md_mod] [94664.084276] md_reap_sync_thread+0x15/0x170 [md_mod] [94664.084292] action_store+0x164/0x2c0 [md_mod] [94664.084309] md_attr_store+0x91/0xe0 [md_mod] [94664.084318] kernfs_fop_write+0x150/0x1d0 [94664.084325] __vfs_write+0x36/0x170 [94664.084330] ? selinux_file_permission+0x117/0x140 [94664.084338] ? security_file_permission+0x3a/0xb0 [94664.084346] vfs_write+0xb9/0x1c0 [94664.084358] SyS_write+0x64/0xe0 [94664.084370] do_syscall_64+0x6e/0x100 [94664.084381] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.084391] RIP: 0033:0x7fee83903970 [94664.084399] RSP: 002b:00007ffd41b1a828 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [94664.084417] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007fee83903970 [94664.084425] RDX: 0000000000000007 RSI: 00000000026d1008 RDI: 0000000000000001 [94664.084431] RBP: 00000000026d1008 R08: 00007fee83bc3760 R09: 00007fee84227700 [94664.084440] R10: 0000000000000073 R11: 0000000000000246 R12: 0000000000000007 [94664.084446] R13: 0000000000000001 R14: 00007fee83bc2600 R15: 0000000000000007 [94664.084461] md0_resync D 0 14081 2 0x80000000 [94664.084478] Call Trace: [94664.084489] ? __schedule+0x282/0x860 [94664.084500] schedule+0x28/0x80 [94664.084521] md_do_sync.cold.80+0x7f1/0x8be [md_mod] [94664.084532] ? finish_wait+0xb0/0xb0 [94664.084540] ? __switch_to_asm+0x40/0x70 [94664.084560] ? find_pers+0x70/0x70 [md_mod] [94664.084579] md_thread+0x9d/0x160 [md_mod] [94664.084588] kthread+0x11a/0x130 [94664.084599] ? __kthread_parkme+0xa0/0xa0 [94664.084610] ret_from_fork+0x22/0x40 [94664.084627] cron-hourly.php D 0 14628 14627 0x00000000 [94664.084642] Call Trace: [94664.084652] ? __schedule+0x282/0x860 [94664.084663] ? __switch_to_asm+0x34/0x70 [94664.084673] schedule+0x28/0x80 [94664.084683] io_schedule+0x12/0x40 [94664.084694] wait_on_page_bit+0x114/0x150 [94664.084704] ? page_cache_tree_insert+0xd0/0xd0 [94664.084715] migrate_pages+0x897/0xb00 [94664.084724] ? isolate_freepages_block+0x3a0/0x3a0 [94664.084733] ? update_pageblock_skip+0xd0/0xd0 [94664.084742] compact_zone+0x5c5/0x860 [94664.084753] ? wake_up_q+0x80/0x80 [94664.084762] compact_node+0xb0/0xf0 [94664.084773] ? lru_add_drain_all_cpuslocked+0x133/0x160 [94664.084782] sysctl_compaction_handler+0x40/0x70 [94664.084793] proc_sys_call_handler+0xbf/0x120 [94664.084802] __vfs_write+0x36/0x170 [94664.084811] ? selinux_file_permission+0x117/0x140 [94664.084820] ? security_file_permission+0x3a/0xb0 [94664.084831] vfs_write+0xb9/0x1c0 [94664.084840] SyS_write+0x64/0xe0 [94664.084845] do_syscall_64+0x6e/0x100 [94664.084850] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.084856] RIP: 0033:0x7f42c7773970 [94664.084858] RSP: 002b:00007fff05e8b518 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [94664.084862] RAX: ffffffffffffffda RBX: 00007f42c4899000 RCX: 00007f42c7773970 [94664.084864] RDX: 0000000000000001 RSI: 00007f42c4862a18 RDI: 0000000000000003 [94664.084868] RBP: 0000000000000001 R08: 000000000000eef8 R09: 00007f42c77c6110 [94664.084870] R10: 79732f636f72702f R11: 0000000000000246 R12: 00007f42c4862a18 [94664.084872] R13: 0000000000000000 R14: 00007f42c48a4400 R15: 0000000000000000 [94664.084879] dd D 0 14755 14620 0x00000004 [94664.084884] Call Trace: [94664.084891] ? __schedule+0x282/0x860 [94664.084897] schedule+0x28/0x80 [94664.084903] io_schedule+0x12/0x40 [94664.084909] generic_file_read_iter+0x3ce/0xa10 [94664.084916] ? page_cache_tree_insert+0xd0/0xd0 [94664.084921] __vfs_read+0x124/0x170 [94664.084926] vfs_read+0xa8/0x170 [94664.084930] SyS_read+0x64/0xe0 [94664.084936] do_syscall_64+0x6e/0x100 [94664.084941] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [94664.084945] RIP: 0033:0x7f44fc0db910 [94664.084948] RSP: 002b:00007ffcf38e6c58 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [94664.084952] RAX: ffffffffffffffda RBX: 000055ed8c18f400 RCX: 00007f44fc0db910 [94664.084955] RDX: 0000000000100000 RSI: 00007f44fbd4e000 RDI: 0000000000000000 [94664.084958] RBP: 0000000000100000 R08: ffffffffffffffff R09: 0000000000000000 [94664.084960] R10: 000000000000037b R11: 0000000000000246 R12: 00007f44fbd4e000 [94664.084964] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
