On Mon, Oct 21, 2024 at 05:20:12PM +0200, Niklas Cassel wrote: > On Mon, Oct 21, 2024 at 02:07:21PM +0200, Niklas Cassel wrote: > > Hello Yi Lai, > > > > On Mon, Oct 21, 2024 at 06:58:59PM +0800, Lai, Yi wrote: > > > Hi Niklas Cassel, > > > > > > Greetings! > > > > > > I used Syzkaller and found that there is INFO: task hung in blk_mq_get_tag in v6.12-rc3 > > > > > > After bisection and the first bad commit is: > > > " > > > e5dd410acb34 ata: libata: Clear DID_TIME_OUT for ATA PT commands with sense data > > > " > > > > It might be that your bisection results are accurate. > > > > However, after looking at the stacktraces, I find it way more likely that > > bisection has landed on the wrong commit. > > > > See this series that was just queued (for 6.13) a few days ago that solves a > > similar starvation: > > https://lore.kernel.org/linux-block/20241014092934.53630-1-songmuchun@xxxxxxxxxxxxx/ > > https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-6.13/block > > > > You could perhaps run with v6.14-rc4 (which should be able to trigger the bug) > > and then try v6.14-rc4 + that series applied, to see if you can still trigger > > the bug? > > Another patch that might be relevant: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e972b08b91ef48488bae9789f03cfedb148667fb > > Which fixes a use after delete in rq_qos_wake_function(). > (We can see that the stack trace has rq_qos_wake_function() before > getting stuck forever in rq_qos_wait()) > > Who knows what could go wrong when accessing a deleted entry, in the > report there was a crash, but I could image other surprises :) > The fix was first included in v6.12-rc4. > > Hi Niklas, Thanks for the info. I have tried using v6.12-rc4 kernel to reproduce the issue. Using the same repro binary, the issue still exists. [ 300.430914] INFO: task kworker/u8:0:11 blocked for more than 147 seconds. [ 300.431593] Not tainted 6.12.0-rc4-kvm #1 [ 300.431988] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.432640] task:kworker/u8:0 state:D stack:0 pid:11 tgid:11 ppid:2 flags:0x00004000 [ 300.433431] Workqueue: writeback wb_workfn (flush-8:0) [ 300.433908] Call Trace: [ 300.434132] <TASK> [ 300.434335] __schedule+0xe13/0x33a0 [ 300.434687] ? __pfx___schedule+0x10/0x10 [ 300.435055] ? lock_release+0x441/0x870 [ 300.435393] ? __pfx_lock_release+0x10/0x10 [ 300.435730] ? trace_lock_acquire+0x139/0x1b0 [ 300.436100] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.436533] ? kthread_data+0x61/0xd0 [ 300.436848] schedule+0xf6/0x3f0 [ 300.437123] io_schedule+0xce/0x150 [ 300.437413] bit_wait_io+0x1f/0x100 [ 300.437707] __wait_on_bit+0x68/0x1b0 [ 300.438011] ? __pfx_bit_wait_io+0x10/0x10 [ 300.438351] out_of_line_wait_on_bit+0xe6/0x120 [ 300.438750] ? __pfx_out_of_line_wait_on_bit+0x10/0x10 [ 300.439162] ? __pfx_wake_bit_function+0x10/0x10 [ 300.439540] ? __might_sleep+0xa3/0x170 [ 300.439864] do_get_write_access+0x83b/0x1010 [ 300.440241] jbd2_journal_get_write_access+0x1e4/0x270 [ 300.440655] __ext4_journal_get_write_access+0x72/0x3b0 [ 300.441238] ext4_mb_mark_context+0x184/0xe30 [ 300.441709] ? __pfx_lock_release+0x10/0x10 [ 300.442156] ? __pfx_ext4_mb_mark_context+0x10/0x10 [ 300.442650] ? ext4_sb_block_valid+0x3a6/0x670 [ 300.443134] ext4_mb_mark_diskspace_used+0x44d/0x9e0 [ 300.443595] ? ext4_get_group_no_and_offset+0x186/0x1f0 [ 300.444074] ? __pfx_ext4_mb_mark_diskspace_used+0x10/0x10 [ 300.444582] ext4_mb_new_blocks+0x779/0x4790 [ 300.444966] ? lock_is_held_type+0xef/0x150 [ 300.445349] ? kasan_save_track+0x18/0x40 [ 300.445727] ? __pfx_ext4_mb_new_blocks+0x10/0x10 [ 300.446139] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.446598] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.447066] ? ext4_inode_to_goal_block+0x340/0x450 [ 300.447486] ext4_ext_map_blocks+0x16f0/0x6060 [ 300.447877] ? __pfx___lock_acquire+0x10/0x10 [ 300.448251] ? stack_trace_save+0x97/0xd0 [ 300.448599] ? __pfx_ext4_ext_map_blocks+0x10/0x10 [ 300.449001] ? debug_smp_processor_id+0x20/0x30 [ 300.449372] ? rcu_is_watching+0x19/0xc0 [ 300.449701] ? trace_lock_acquire+0x139/0x1b0 [ 300.450065] ? lock_acquire+0x80/0xb0 [ 300.450359] ? ext4_map_blocks+0x344/0x1390 [ 300.450696] ? __pfx_down_write+0x10/0x10 [ 300.451048] ext4_map_blocks+0x421/0x1390 [ 300.451373] ? __pfx_ext4_map_blocks+0x10/0x10 [ 300.451718] ? kasan_save_track+0x18/0x40 [ 300.452030] ? debug_smp_processor_id+0x20/0x30 [ 300.452374] ? rcu_is_watching+0x19/0xc0 [ 300.452674] ? kmem_cache_alloc_noprof+0x2e7/0x370 [ 300.453040] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.453430] ext4_do_writepages+0x18e0/0x2f30 [ 300.453775] ? __pfx_ext4_do_writepages+0x10/0x10 [ 300.454115] ? rcu_is_watching+0x19/0xc0 [ 300.454411] ? __this_cpu_preempt_check+0x21/0x30 [ 300.454760] ? lock_is_held_type+0xef/0x150 [ 300.455063] ext4_writepages+0x35c/0x7c0 [ 300.455345] ? __pfx_ext4_writepages+0x10/0x10 [ 300.455658] ? lock_release+0x441/0x870 [ 300.455948] ? do_raw_spin_unlock+0x15c/0x210 [ 300.456250] ? __pfx_ext4_writepages+0x10/0x10 [ 300.456557] do_writepages+0x1c3/0x8a0 [ 300.456833] ? __pfx_do_writepages+0x10/0x10 [ 300.457127] ? __pfx_lock_release+0x10/0x10 [ 300.457413] ? __pfx_lock_release+0x10/0x10 [ 300.457711] __writeback_single_inode+0x124/0xe10 [ 300.458033] ? wbc_attach_and_unlock_inode+0x5a9/0x950 [ 300.458385] writeback_sb_inodes+0x63b/0x1000 [ 300.458675] ? lock_release+0x441/0x870 [ 300.458956] ? __pfx_writeback_sb_inodes+0x10/0x10 [ 300.459270] ? lock_acquire.part.0+0x152/0x390 [ 300.459605] ? __sanitizer_cov_trace_const_cmp8+0x1c/0x30 [ 300.459959] __writeback_inodes_wb+0xc8/0x280 [ 300.460248] wb_writeback+0x717/0xaf0 [ 300.460496] ? __pfx_wb_writeback+0x10/0x10 [ 300.460772] ? __sanitizer_cov_trace_cmp4+0x1a/0x20 [ 300.461086] ? get_nr_dirty_inodes+0x12c/0x1b0 [ 300.461374] ? lockdep_hardirqs_on+0x89/0x110 [ 300.461660] wb_workfn+0x7b5/0xc90 [ 300.461888] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 300.462198] ? __pfx_wb_workfn+0x10/0x10 [ 300.462446] ? rcu_is_watching+0x19/0xc0 [ 300.462707] process_one_work+0x92e/0x1b50 [ 300.462994] ? lock_acquire.part.0+0x152/0x390 [ 300.463285] ? __pfx_process_one_work+0x10/0x10 [ 300.463566] ? move_linked_works+0x1bf/0x2c0 [ 300.463839] ? __this_cpu_preempt_check+0x21/0x30 [ 300.464138] ? assign_work+0x19f/0x250 [ 300.464375] ? lock_is_held_type+0xef/0x150 [ 300.464641] worker_thread+0x68d/0xe90 [ 300.464893] kthread+0x35a/0x470 [ 300.465095] ? __pfx_worker_thread+0x10/0x10 [ 300.465355] ? __pfx_kthread+0x10/0x10 [ 300.465587] ret_from_fork+0x56/0x90 [ 300.465812] ? __pfx_kthread+0x10/0x10 [ 300.466043] ret_from_fork_asm+0x1a/0x30 [ 300.466297] </TASK> [ 300.466451] INFO: task kworker/u8:1:31 blocked for more than 147 seconds. [ 300.466858] Not tainted 6.12.0-rc4-kvm #1 [ 300.467127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.467573] task:kworker/u8:1 state:D stack:0 pid:31 tgid:31 ppid:2 flags:0x00004000 [ 300.468115] Workqueue: writeback wb_workfn (flush-8:0) [ 300.468426] Call Trace: [ 300.468579] <TASK> [ 300.468717] __schedule+0xe13/0x33a0 [ 300.468951] ? __pfx___schedule+0x10/0x10 [ 300.469194] ? lock_release+0x441/0x870 [ 300.469433] ? __pfx_lock_release+0x10/0x10 [ 300.469686] ? trace_lock_acquire+0x139/0x1b0 [ 300.469957] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.470270] ? kthread_data+0x61/0xd0 [ 300.470492] schedule+0xf6/0x3f0 [ 300.470692] io_schedule+0xce/0x150 [ 300.470917] blk_mq_get_tag+0x5e7/0xce0 [ 300.471160] ? __pfx_blk_mq_get_tag+0x10/0x10 [ 300.471417] ? __pfx_wbt_cleanup_cb+0x10/0x10 [ 300.471677] ? __sanitizer_cov_trace_switch+0x58/0xa0 [ 300.471971] ? __pfx_autoremove_wake_function+0x10/0x10 [ 300.472275] ? __pfx_rq_qos_wake_function+0x10/0x10 [ 300.472558] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.472863] ? dd_limit_depth+0x21e/0x2a0 [ 300.473098] ? __pfx_dd_limit_depth+0x10/0x10 [ 300.473349] __blk_mq_alloc_requests+0x6f6/0x1720 [ 300.473632] ? __pfx___blk_mq_alloc_requests+0x10/0x10 [ 300.473926] ? __pfx_wbt_wait+0x10/0x10 [ 300.474152] blk_mq_submit_bio+0x93c/0x2a20 [ 300.474391] ? __kasan_check_read+0x15/0x20 [ 300.474645] ? __pfx_blk_mq_submit_bio+0x10/0x10 [ 300.474921] ? __pfx_mark_lock.part.0+0x10/0x10 [ 300.475182] ? __this_cpu_preempt_check+0x21/0x30 [ 300.475457] ? __this_cpu_preempt_check+0x21/0x30 [ 300.475727] ? lock_release+0x441/0x870 [ 300.475953] __submit_bio+0x18d/0x500 [ 300.476169] ? __pfx___submit_bio+0x10/0x10 [ 300.476407] ? __this_cpu_preempt_check+0x21/0x30 [ 300.476668] ? seqcount_lockdep_reader_access.constprop.0+0xb4/0xd0 [ 300.477018] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.477313] ? kvm_clock_get_cycles+0x43/0x70 [ 300.477566] submit_bio_noacct_nocheck+0x6a0/0xcc0 [ 300.477834] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10 [ 300.478127] ? lock_is_held_type+0xef/0x150 [ 300.478369] submit_bio_noacct+0x61d/0x1dc0 [ 300.478610] submit_bio+0xce/0x480 [ 300.478819] ext4_io_submit+0xb0/0x150 [ 300.479036] ext4_do_writepages+0xa54/0x2f30 [ 300.479284] ? __this_cpu_preempt_check+0x21/0x30 [ 300.479556] ? __pfx_ext4_do_writepages+0x10/0x10 [ 300.479817] ? rcu_is_watching+0x19/0xc0 [ 300.480042] ? __this_cpu_preempt_check+0x21/0x30 [ 300.480304] ? lock_is_held_type+0xef/0x150 [ 300.480544] ext4_writepages+0x35c/0x7c0 [ 300.480766] ? __pfx_ext4_writepages+0x10/0x10 [ 300.481019] ? lock_release+0x441/0x870 [ 300.481247] ? do_raw_spin_unlock+0x15c/0x210 [ 300.481491] ? __pfx_ext4_writepages+0x10/0x10 [ 300.481734] do_writepages+0x1c3/0x8a0 [ 300.481947] ? __pfx_do_writepages+0x10/0x10 [ 300.482181] ? __pfx_lock_release+0x10/0x10 [ 300.482409] ? __pfx_lock_release+0x10/0x10 [ 300.482646] __writeback_single_inode+0x124/0xe10 [ 300.482912] ? wbc_attach_and_unlock_inode+0x5a9/0x950 [ 300.483192] writeback_sb_inodes+0x63b/0x1000 [ 300.483431] ? lock_release+0x441/0x870 [ 300.483652] ? __pfx_writeback_sb_inodes+0x10/0x10 [ 300.483911] ? lock_acquire.part.0+0x152/0x390 [ 300.484187] ? __sanitizer_cov_trace_const_cmp8+0x1c/0x30 [ 300.484478] __writeback_inodes_wb+0xc8/0x280 [ 300.484724] wb_writeback+0x717/0xaf0 [ 300.484938] ? __pfx_wb_writeback+0x10/0x10 [ 300.485172] ? __sanitizer_cov_trace_cmp4+0x1a/0x20 [ 300.485433] ? get_nr_dirty_inodes+0x12c/0x1b0 [ 300.485675] ? lockdep_hardirqs_on+0x89/0x110 [ 300.485917] wb_workfn+0x7b5/0xc90 [ 300.486111] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 300.486375] ? __pfx_wb_workfn+0x10/0x10 [ 300.486591] ? rcu_is_watching+0x19/0xc0 [ 300.486831] process_one_work+0x92e/0x1b50 [ 300.487055] ? lock_acquire.part.0+0x152/0x390 [ 300.487308] ? __pfx_process_one_work+0x10/0x10 [ 300.487555] ? move_linked_works+0x1bf/0x2c0 [ 300.487791] ? __this_cpu_preempt_check+0x21/0x30 [ 300.488053] ? assign_work+0x19f/0x250 [ 300.488260] ? lock_is_held_type+0xef/0x150 [ 300.488493] worker_thread+0x68d/0xe90 [ 300.488701] ? lockdep_hardirqs_on+0x89/0x110 [ 300.488960] kthread+0x35a/0x470 [ 300.489141] ? __pfx_worker_thread+0x10/0x10 [ 300.489377] ? __pfx_kthread+0x10/0x10 [ 300.489586] ret_from_fork+0x56/0x90 [ 300.489785] ? __pfx_kthread+0x10/0x10 [ 300.489993] ret_from_fork_asm+0x1a/0x30 [ 300.490221] </TASK> [ 300.490347] INFO: task kworker/u8:2:35 blocked for more than 147 seconds. [ 300.490698] Not tainted 6.12.0-rc4-kvm #1 [ 300.490952] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.491353] task:kworker/u8:2 state:D stack:0 pid:35 tgid:35 ppid:2 flags:0x00004000 [ 300.491835] Workqueue: writeback wb_workfn (flush-8:0) [ 300.492115] Call Trace: [ 300.492252] <TASK> [ 300.492377] __schedule+0xe13/0x33a0 [ 300.492585] ? __pfx___schedule+0x10/0x10 [ 300.492811] ? lock_release+0x441/0x870 [ 300.493031] ? __pfx_lock_release+0x10/0x10 [ 300.493259] ? trace_lock_acquire+0x139/0x1b0 [ 300.493502] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.493784] ? kthread_data+0x61/0xd0 [ 300.493985] schedule+0xf6/0x3f0 [ 300.494165] io_schedule+0xce/0x150 [ 300.494358] blk_mq_get_tag+0x5e7/0xce0 [ 300.494570] ? __pfx_blk_mq_get_tag+0x10/0x10 [ 300.494815] ? __pfx___mod_timer+0x10/0x10 [ 300.495036] ? __pfx_autoremove_wake_function+0x10/0x10 [ 300.495311] ? __pfx_rq_qos_wake_function+0x10/0x10 [ 300.495571] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.495852] ? dd_limit_depth+0x21e/0x2a0 [ 300.496069] ? __pfx_dd_limit_depth+0x10/0x10 [ 300.496303] __blk_mq_alloc_requests+0x6f6/0x1720 [ 300.496552] ? mod_timer+0x29/0x40 [ 300.496749] ? __pfx___blk_mq_alloc_requests+0x10/0x10 [ 300.497028] ? __pfx_wbt_wait+0x10/0x10 [ 300.497239] blk_mq_submit_bio+0x93c/0x2a20 [ 300.497461] ? __kasan_check_read+0x15/0x20 [ 300.497691] ? __pfx_blk_mq_submit_bio+0x10/0x10 [ 300.497935] ? __pfx_mark_lock.part.0+0x10/0x10 [ 300.498177] ? __this_cpu_preempt_check+0x21/0x30 [ 300.498433] ? __this_cpu_preempt_check+0x21/0x30 [ 300.498683] ? lock_release+0x441/0x870 [ 300.498905] __submit_bio+0x18d/0x500 [ 300.499105] ? __pfx___submit_bio+0x10/0x10 [ 300.499329] ? __this_cpu_preempt_check+0x21/0x30 [ 300.499578] ? seqcount_lockdep_reader_access.constprop.0+0xb4/0xd0 [ 300.499908] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.500192] ? kvm_clock_get_cycles+0x43/0x70 [ 300.500430] submit_bio_noacct_nocheck+0x6a0/0xcc0 [ 300.500687] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10 [ 300.500985] submit_bio_noacct+0x61d/0x1dc0 [ 300.501216] submit_bio+0xce/0x480 [ 300.501405] ext4_io_submit+0xb0/0x150 [ 300.501611] ext4_do_writepages+0x1103/0x2f30 [ 300.501862] ? __pfx_ext4_do_writepages+0x10/0x10 [ 300.502111] ? rcu_is_watching+0x19/0xc0 [ 300.502326] ? __this_cpu_preempt_check+0x21/0x30 [ 300.502577] ? lock_is_held_type+0xef/0x150 [ 300.502820] ext4_writepages+0x35c/0x7c0 [ 300.503034] ? __pfx_ext4_writepages+0x10/0x10 [ 300.503271] ? lock_release+0x441/0x870 [ 300.503489] ? do_raw_spin_unlock+0x15c/0x210 [ 300.503725] ? __pfx_ext4_writepages+0x10/0x10 [ 300.503964] do_writepages+0x1c3/0x8a0 [ 300.504171] ? __pfx_do_writepages+0x10/0x10 [ 300.504400] ? __pfx_lock_release+0x10/0x10 [ 300.504623] ? __pfx_lock_release+0x10/0x10 [ 300.504865] __writeback_single_inode+0x124/0xe10 [ 300.505117] ? wbc_attach_and_unlock_inode+0x5a9/0x950 [ 300.505399] writeback_sb_inodes+0x63b/0x1000 [ 300.505641] ? lock_release+0x441/0x870 [ 300.505863] ? __pfx_writeback_sb_inodes+0x10/0x10 [ 300.506123] ? lock_acquire.part.0+0x152/0x390 [ 300.506400] ? __sanitizer_cov_trace_const_cmp8+0x1c/0x30 [ 300.506692] __writeback_inodes_wb+0xc8/0x280 [ 300.506953] wb_writeback+0x717/0xaf0 [ 300.507164] ? __pfx_wb_writeback+0x10/0x10 [ 300.507394] ? __sanitizer_cov_trace_cmp4+0x1a/0x20 [ 300.507656] ? get_nr_dirty_inodes+0x12c/0x1b0 [ 300.507898] ? lockdep_hardirqs_on+0x89/0x110 [ 300.508140] wb_workfn+0x7b5/0xc90 [ 300.508337] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 300.508601] ? __pfx_wb_workfn+0x10/0x10 [ 300.508822] ? rcu_is_watching+0x19/0xc0 [ 300.509047] process_one_work+0x92e/0x1b50 [ 300.509265] ? lock_acquire.part.0+0x152/0x390 [ 300.509512] ? __pfx_process_one_work+0x10/0x10 [ 300.509752] ? move_linked_works+0x1bf/0x2c0 [ 300.509982] ? __this_cpu_preempt_check+0x21/0x30 [ 300.510238] ? assign_work+0x19f/0x250 [ 300.510440] ? lock_is_held_type+0xef/0x150 [ 300.510667] worker_thread+0x68d/0xe90 [ 300.510895] kthread+0x35a/0x470 [ 300.511073] ? __pfx_worker_thread+0x10/0x10 [ 300.511302] ? __pfx_kthread+0x10/0x10 [ 300.511507] ret_from_fork+0x56/0x90 [ 300.511702] ? __pfx_kthread+0x10/0x10 [ 300.511906] ret_from_fork_asm+0x1a/0x30 [ 300.512128] </TASK> [ 300.512270] INFO: task jbd2/sda-8:104 blocked for more than 147 seconds. [ 300.512609] Not tainted 6.12.0-rc4-kvm #1 [ 300.512847] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.513237] task:jbd2/sda-8 state:D stack:0 pid:104 tgid:104 ppid:2 flags:0x00004000 [ 300.513707] Call Trace: [ 300.513840] <TASK> [ 300.513961] __schedule+0xe13/0x33a0 [ 300.514162] ? __pfx___schedule+0x10/0x10 [ 300.514376] ? lock_release+0x441/0x870 [ 300.514586] ? __pfx_lock_release+0x10/0x10 [ 300.514819] ? trace_lock_acquire+0x139/0x1b0 [ 300.515057] ? lock_acquire+0x80/0xb0 [ 300.515254] ? schedule+0x216/0x3f0 [ 300.515448] schedule+0xf6/0x3f0 [ 300.515629] io_schedule+0xce/0x150 [ 300.515820] blk_mq_get_tag+0x5e7/0xce0 [ 300.516033] ? __pfx_blk_mq_get_tag+0x10/0x10 [ 300.516266] ? lock_release+0x441/0x870 [ 300.516471] ? __sanitizer_cov_trace_switch+0x58/0xa0 [ 300.516737] ? __pfx_autoremove_wake_function+0x10/0x10 [ 300.517014] ? blk_mq_sched_try_merge+0xce/0x650 [ 300.517261] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.517542] ? dd_limit_depth+0x21e/0x2a0 [ 300.517758] ? __pfx_dd_limit_depth+0x10/0x10 [ 300.517992] __blk_mq_alloc_requests+0x6f6/0x1720 [ 300.518246] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.518534] ? __pfx___blk_mq_alloc_requests+0x10/0x10 [ 300.518818] ? __pfx_wbt_wait+0x10/0x10 [ 300.519029] blk_mq_submit_bio+0x93c/0x2a20 [ 300.519251] ? __pfx___lock_acquire+0x10/0x10 [ 300.519490] ? __pfx_blk_mq_submit_bio+0x10/0x10 [ 300.519736] ? __this_cpu_preempt_check+0x21/0x30 [ 300.519991] ? __this_cpu_preempt_check+0x21/0x30 [ 300.520242] ? lock_release+0x441/0x870 [ 300.520451] __submit_bio+0x18d/0x500 [ 300.520650] ? __pfx___submit_bio+0x10/0x10 [ 300.520879] ? __this_cpu_preempt_check+0x21/0x30 [ 300.521130] ? seqcount_lockdep_reader_access.constprop.0+0xb4/0xd0 [ 300.521458] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.521738] ? kvm_clock_get_cycles+0x43/0x70 [ 300.521977] submit_bio_noacct_nocheck+0x6a0/0xcc0 [ 300.522226] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10 [ 300.522504] ? __sanitizer_cov_trace_switch+0x58/0xa0 [ 300.522782] submit_bio_noacct+0x61d/0x1dc0 [ 300.523006] submit_bio+0xce/0x480 [ 300.523190] submit_bh_wbc+0x58a/0x740 [ 300.523397] submit_bh+0x2b/0x40 [ 300.523572] jbd2_journal_commit_transaction+0x2a09/0x65e0 [ 300.523873] ? __pfx_jbd2_journal_commit_transaction+0x10/0x10 [ 300.524176] ? __this_cpu_preempt_check+0x21/0x30 [ 300.524420] ? lock_release+0x441/0x870 [ 300.524630] ? __this_cpu_preempt_check+0x21/0x30 [ 300.524879] ? _raw_spin_unlock_irqrestore+0x45/0x70 [ 300.525135] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.525417] ? kjournald2+0x5e7/0x790 [ 300.525616] kjournald2+0x204/0x790 [ 300.525809] ? __pfx_kjournald2+0x10/0x10 [ 300.526017] ? lockdep_hardirqs_on+0x89/0x110 [ 300.526245] ? __pfx_autoremove_wake_function+0x10/0x10 [ 300.526513] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.526797] ? __kthread_parkme+0x15d/0x230 [ 300.527019] kthread+0x35a/0x470 [ 300.527192] ? __pfx_kjournald2+0x10/0x10 [ 300.527399] ? __pfx_kthread+0x10/0x10 [ 300.527597] ret_from_fork+0x56/0x90 [ 300.527787] ? __pfx_kthread+0x10/0x10 [ 300.527984] ret_from_fork_asm+0x1a/0x30 [ 300.528200] </TASK> [ 300.528319] INFO: task kworker/u8:4:167 blocked for more than 147 seconds. [ 300.528657] Not tainted 6.12.0-rc4-kvm #1 [ 300.528891] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.529273] task:kworker/u8:4 state:D stack:0 pid:167 tgid:167 ppid:2 flags:0x00004000 [ 300.529732] Workqueue: writeback wb_workfn (flush-8:0) [ 300.529998] Call Trace: [ 300.530128] <TASK> [ 300.530246] __schedule+0xe13/0x33a0 [ 300.530443] ? __pfx___schedule+0x10/0x10 [ 300.530651] ? lock_release+0x441/0x870 [ 300.530867] ? __pfx_lock_release+0x10/0x10 [ 300.531084] ? trace_lock_acquire+0x139/0x1b0 [ 300.531316] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.531589] ? kthread_data+0x61/0xd0 [ 300.531785] schedule+0xf6/0x3f0 [ 300.531961] io_schedule+0xce/0x150 [ 300.532148] rq_qos_wait+0x1c5/0x310 [ 300.532336] ? __pfx_wbt_cleanup_cb+0x10/0x10 [ 300.532563] ? __pfx_rq_qos_wait+0x10/0x10 [ 300.532780] ? __pfx_lock_release+0x10/0x10 [ 300.533000] ? __pfx_rq_qos_wake_function+0x10/0x10 [ 300.533254] ? __pfx_wbt_inflight_cb+0x10/0x10 [ 300.533487] ? do_raw_spin_unlock+0x15c/0x210 [ 300.533719] wbt_wait+0x1ec/0x400 [ 300.533898] ? __pfx_wbt_wait+0x10/0x10 [ 300.534100] ? __pfx_dd_bio_merge+0x10/0x10 [ 300.534321] ? __pfx_wbt_wait+0x10/0x10 [ 300.534523] __rq_qos_throttle+0x5f/0xb0 [ 300.534748] blk_mq_submit_bio+0x8ad/0x2a20 [ 300.534966] ? __kasan_check_read+0x15/0x20 [ 300.535188] ? __pfx_blk_mq_submit_bio+0x10/0x10 [ 300.535426] ? __pfx_mark_lock.part.0+0x10/0x10 [ 300.535663] ? __this_cpu_preempt_check+0x21/0x30 [ 300.535912] ? __this_cpu_preempt_check+0x21/0x30 [ 300.536156] ? lock_release+0x441/0x870 [ 300.536362] __submit_bio+0x18d/0x500 [ 300.536556] ? __pfx___submit_bio+0x10/0x10 [ 300.536777] ? __this_cpu_preempt_check+0x21/0x30 [ 300.537021] ? seqcount_lockdep_reader_access.constprop.0+0xb4/0xd0 [ 300.537342] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.537616] ? kvm_clock_get_cycles+0x43/0x70 [ 300.537849] submit_bio_noacct_nocheck+0x6a0/0xcc0 [ 300.538098] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10 [ 300.538375] ? __sanitizer_cov_trace_switch+0x58/0xa0 [ 300.538637] submit_bio_noacct+0x61d/0x1dc0 [ 300.538874] submit_bio+0xce/0x480 [ 300.539059] submit_bh_wbc+0x58a/0x740 [ 300.539263] __block_write_full_folio+0x7a1/0xed0 [ 300.539506] ? __pfx_blkdev_get_block+0x10/0x10 [ 300.539749] block_write_full_folio+0x2e6/0x3a0 [ 300.539985] ? __pfx_blkdev_get_block+0x10/0x10 [ 300.540224] ? __pfx_blkdev_get_block+0x10/0x10 [ 300.540460] write_cache_pages+0xbc/0x140 [ 300.540670] ? __pfx_block_write_full_folio+0x10/0x10 [ 300.540933] ? __pfx_write_cache_pages+0x10/0x10 [ 300.541178] ? __this_cpu_preempt_check+0x21/0x30 [ 300.541427] blkdev_writepages+0xa5/0xf0 [ 300.541634] ? __pfx_blkdev_writepages+0x10/0x10 [ 300.541874] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 300.542117] ? do_raw_spin_unlock+0x15c/0x210 [ 300.542349] ? __pfx_blkdev_writepages+0x10/0x10 [ 300.542590] do_writepages+0x1c3/0x8a0 [ 300.542809] ? __pfx_do_writepages+0x10/0x10 [ 300.543017] ? __pfx_lock_release+0x10/0x10 [ 300.543216] ? lock_release+0x441/0x870 [ 300.543410] __writeback_single_inode+0x124/0xe10 [ 300.543649] ? wbc_attach_and_unlock_inode+0x5a9/0x950 [ 300.543918] writeback_sb_inodes+0x63b/0x1000 [ 300.544159] ? __pfx_writeback_sb_inodes+0x10/0x10 [ 300.544405] ? lock_acquire.part.0+0x152/0x390 [ 300.544669] ? __sanitizer_cov_trace_const_cmp8+0x1c/0x30 [ 300.544949] __writeback_inodes_wb+0xc8/0x280 [ 300.545184] wb_writeback+0x717/0xaf0 [ 300.545385] ? __pfx_wb_writeback+0x10/0x10 [ 300.545604] ? __sanitizer_cov_trace_cmp4+0x1a/0x20 [ 300.545859] ? __sanitizer_cov_trace_cmp4+0x1a/0x20 [ 300.546109] ? get_nr_dirty_inodes+0x12c/0x1b0 [ 300.546342] ? lockdep_hardirqs_on+0x89/0x110 [ 300.546574] wb_workfn+0x76c/0xc90 [ 300.546773] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 300.547026] ? __pfx_wb_workfn+0x10/0x10 [ 300.547234] ? rcu_is_watching+0x19/0xc0 [ 300.547452] process_one_work+0x92e/0x1b50 [ 300.547666] ? lock_acquire.part.0+0x152/0x390 [ 300.547908] ? __pfx_process_one_work+0x10/0x10 [ 300.548144] ? move_linked_works+0x1bf/0x2c0 [ 300.548370] ? __this_cpu_preempt_check+0x21/0x30 [ 300.548620] ? assign_work+0x19f/0x250 [ 300.548821] ? lock_is_held_type+0xef/0x150 [ 300.549043] worker_thread+0x68d/0xe90 [ 300.549242] ? lockdep_hardirqs_on+0x89/0x110 [ 300.549481] kthread+0x35a/0x470 [ 300.549655] ? __pfx_worker_thread+0x10/0x10 [ 300.549880] ? __pfx_kthread+0x10/0x10 [ 300.550080] ret_from_fork+0x56/0x90 [ 300.550270] ? __pfx_kthread+0x10/0x10 [ 300.550470] ret_from_fork_asm+0x1a/0x30 [ 300.550687] </TASK> [ 300.550826] INFO: task kworker/1:3:233 blocked for more than 147 seconds. [ 300.551165] Not tainted 6.12.0-rc4-kvm #1 [ 300.551396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.551780] task:kworker/1:3 state:D stack:0 pid:233 tgid:233 ppid:2 flags:0x00004000 [ 300.552242] Workqueue: events ata_scsi_dev_rescan [ 300.552501] Call Trace: [ 300.552635] <TASK> [ 300.552754] __schedule+0xe13/0x33a0 [ 300.552955] ? __pfx___schedule+0x10/0x10 [ 300.553165] ? lock_release+0x441/0x870 [ 300.553370] ? __pfx_lock_release+0x10/0x10 [ 300.553594] ? _raw_spin_unlock_irq+0x3c/0x60 [ 300.553825] schedule+0xf6/0x3f0 [ 300.554002] io_schedule+0xce/0x150 [ 300.554190] blk_mq_get_tag+0x5e7/0xce0 [ 300.554398] ? __pfx_blk_mq_get_tag+0x10/0x10 [ 300.554629] ? __pfx_autoremove_wake_function+0x10/0x10 [ 300.554934] __blk_mq_alloc_requests+0x6f6/0x1720 [ 300.555181] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.555466] ? __pfx___blk_mq_alloc_requests+0x10/0x10 [ 300.555740] blk_mq_alloc_request+0x655/0x930 [ 300.555973] ? __pfx_blk_mq_alloc_request+0x10/0x10 [ 300.556225] ? __lock_acquire+0xd87/0x5c90 [ 300.556450] scsi_execute_cmd+0x223/0x1040 [ 300.556684] ? __this_cpu_preempt_check+0x21/0x30 [ 300.556933] ? __pfx_scsi_execute_cmd+0x10/0x10 [ 300.557178] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 300.557431] scsi_vpd_inquiry+0xdf/0x220 [ 300.557641] ? __pfx_scsi_vpd_inquiry+0x10/0x10 [ 300.557877] ? debug_smp_processor_id+0x20/0x30 [ 300.558113] ? rcu_is_watching+0x19/0xc0 [ 300.558325] scsi_get_vpd_size+0x11e/0x310 [ 300.558539] ? __mutex_lock+0x230/0x1490 [ 300.558763] ? __pfx_scsi_get_vpd_size+0x10/0x10 [ 300.559004] ? scsi_rescan_device+0x33/0x360 [ 300.559227] ? __pfx_mark_lock.part.0+0x10/0x10 [ 300.559465] ? __this_cpu_preempt_check+0x21/0x30 [ 300.559713] ? lock_acquire.part.0+0x152/0x390 [ 300.559947] ? __pfx___mutex_lock+0x10/0x10 [ 300.560172] scsi_get_vpd_buf+0x30/0x180 [ 300.560382] scsi_attach_vpd+0xf8/0x5b0 [ 300.560591] scsi_rescan_device+0x106/0x360 [ 300.560820] ata_scsi_dev_rescan+0x1d2/0x470 [ 300.561051] process_one_work+0x92e/0x1b50 [ 300.561264] ? lock_acquire.part.0+0x152/0x390 [ 300.561505] ? __pfx_process_one_work+0x10/0x10 [ 300.561741] ? move_linked_works+0x1bf/0x2c0 [ 300.561972] ? __this_cpu_preempt_check+0x21/0x30 [ 300.562220] ? assign_work+0x19f/0x250 [ 300.562421] ? lock_is_held_type+0xef/0x150 [ 300.562645] worker_thread+0x68d/0xe90 [ 300.562856] ? lockdep_hardirqs_on+0x89/0x110 [ 300.563105] kthread+0x35a/0x470 [ 300.563278] ? __pfx_worker_thread+0x10/0x10 [ 300.563501] ? __pfx_kthread+0x10/0x10 [ 300.563700] ret_from_fork+0x56/0x90 [ 300.563892] ? __pfx_kthread+0x10/0x10 [ 300.564093] ret_from_fork_asm+0x1a/0x30 [ 300.564313] </TASK> [ 300.564432] INFO: task auditd:240 blocked for more than 147 seconds. [ 300.564749] Not tainted 6.12.0-rc4-kvm #1 [ 300.564984] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.565375] task:auditd state:D stack:0 pid:240 tgid:240 ppid:1 flags:0x00000002 [ 300.565832] Call Trace: [ 300.565963] <TASK> [ 300.566084] __schedule+0xe13/0x33a0 [ 300.566284] ? __pfx___schedule+0x10/0x10 [ 300.566495] ? lock_release+0x441/0x870 [ 300.566700] ? __pfx_lock_release+0x10/0x10 [ 300.566932] ? trace_lock_acquire+0x139/0x1b0 [ 300.567165] ? lock_acquire+0x80/0xb0 [ 300.567362] ? schedule+0x216/0x3f0 [ 300.567553] schedule+0xf6/0x3f0 [ 300.567730] io_schedule+0xce/0x150 [ 300.567921] bit_wait_io+0x1f/0x100 [ 300.568110] __wait_on_bit+0x68/0x1b0 [ 300.568307] ? __pfx_bit_wait_io+0x10/0x10 [ 300.568528] out_of_line_wait_on_bit+0xe6/0x120 [ 300.568775] ? __pfx_out_of_line_wait_on_bit+0x10/0x10 [ 300.569042] ? __pfx_wake_bit_function+0x10/0x10 [ 300.569286] ? __might_sleep+0xa3/0x170 [ 300.569495] do_get_write_access+0x83b/0x1010 [ 300.569733] jbd2_journal_get_write_access+0x1e4/0x270 [ 300.570003] __ext4_journal_get_write_access+0x72/0x3b0 [ 300.570276] ext4_reserve_inode_write+0x199/0x280 [ 300.570522] ? ext4_dirty_inode+0xeb/0x140 [ 300.570754] __ext4_mark_inode_dirty+0x18b/0x8c0 [ 300.570999] ? debug_smp_processor_id+0x20/0x30 [ 300.571238] ? __pfx___ext4_mark_inode_dirty+0x10/0x10 [ 300.571502] ? trace_jbd2_handle_start+0x77/0x210 [ 300.571750] ? jbd2__journal_start+0xff/0x6b0 [ 300.571986] ? __ext4_journal_start_sb+0x1f8/0x600 [ 300.572237] ? ext4_dirty_inode+0xb3/0x140 [ 300.572460] ? __pfx_ext4_dirty_inode+0x10/0x10 [ 300.572702] ext4_dirty_inode+0xeb/0x140 [ 300.572911] ? rcu_is_watching+0x19/0xc0 [ 300.573121] __mark_inode_dirty+0x1d0/0xd50 [ 300.573343] ? __pfx_generic_write_checks+0x10/0x10 [ 300.573608] generic_update_time+0xe0/0x110 [ 300.573832] file_modified+0x1af/0x1e0 [ 300.574039] ext4_buffered_write_iter+0x101/0x3e0 [ 300.574292] ext4_file_write_iter+0x8b8/0x1ab0 [ 300.574524] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 300.574791] ? debug_smp_processor_id+0x20/0x30 [ 300.575035] ? __pfx_ext4_file_write_iter+0x10/0x10 [ 300.575288] ? __this_cpu_preempt_check+0x21/0x30 [ 300.575531] ? lock_is_held_type+0xef/0x150 [ 300.575758] vfs_write+0xc46/0x1120 [ 300.575950] ? __pfx_ext4_file_write_iter+0x10/0x10 [ 300.576211] ? __pfx_vfs_write+0x10/0x10 [ 300.576429] ? mutex_lock_nested+0x1f/0x30 [ 300.576647] ksys_write+0x14f/0x290 [ 300.576839] ? __pfx_ksys_write+0x10/0x10 [ 300.577058] __x64_sys_write+0x7b/0xc0 [ 300.577256] ? syscall_trace_enter+0x9b/0x230 [ 300.577490] x64_sys_call+0x1685/0x20d0 [ 300.577692] do_syscall_64+0x6d/0x140 [ 300.577889] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 300.578156] RIP: 0033:0x7fe1ebf3ebcf [ 300.578348] RSP: 002b:00007ffebc24a5e0 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 [ 300.578741] RAX: ffffffffffffffda RBX: 00000000000000e0 RCX: 00007fe1ebf3ebcf [ 300.579094] RDX: 00000000000000e0 RSI: 0000559099024040 RDI: 0000000000000005 [ 300.579448] RBP: 0000559099024040 R08: 0000000000000000 R09: 00007fe1ebfb14e0 [ 300.579802] R10: 00007fe1ebfb13e0 R11: 0000000000000293 R12: 00000000000000e0 [ 300.580154] R13: 000055909901a2a0 R14: 00000000000000e0 R15: 00007fe1ebff69e0 [ 300.580518] </TASK> [ 300.580641] INFO: task in:imjournal:319 blocked for more than 147 seconds. [ 300.580981] Not tainted 6.12.0-rc4-kvm #1 [ 300.581211] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.581591] task:in:imjournal state:D stack:0 pid:319 tgid:300 ppid:1 flags:0x00000002 [ 300.582047] Call Trace: [ 300.582177] <TASK> [ 300.582296] __schedule+0xe13/0x33a0 [ 300.582492] ? __pfx___schedule+0x10/0x10 [ 300.582700] ? lock_release+0x441/0x870 [ 300.582918] ? __pfx_lock_release+0x10/0x10 [ 300.583135] ? trace_lock_acquire+0x139/0x1b0 [ 300.583366] ? lock_acquire+0x80/0xb0 [ 300.583558] ? schedule+0x216/0x3f0 [ 300.583747] schedule+0xf6/0x3f0 [ 300.583923] io_schedule+0xce/0x150 [ 300.584109] bit_wait_io+0x1f/0x100 [ 300.584297] __wait_on_bit+0x68/0x1b0 [ 300.584491] ? __pfx_bit_wait_io+0x10/0x10 [ 300.584710] out_of_line_wait_on_bit+0xe6/0x120 [ 300.584950] ? __pfx_out_of_line_wait_on_bit+0x10/0x10 [ 300.585216] ? __pfx_wake_bit_function+0x10/0x10 [ 300.585457] ? __might_sleep+0xa3/0x170 [ 300.585662] do_get_write_access+0x83b/0x1010 [ 300.585899] jbd2_journal_get_write_access+0x1e4/0x270 [ 300.586166] __ext4_journal_get_write_access+0x72/0x3b0 [ 300.586436] __ext4_new_inode+0xd66/0x5320 [ 300.586661] ? __pfx___ext4_new_inode+0x10/0x10 [ 300.586910] ? __pfx___dquot_initialize+0x10/0x10 [ 300.587163] ? _raw_spin_unlock+0x31/0x60 [ 300.587372] ? d_splice_alias+0x4e8/0xe30 [ 300.587585] ext4_create+0x32e/0x550 [ 300.587783] ? __pfx_ext4_create+0x10/0x10 [ 300.588000] ? inode_permission+0xf4/0x650 [ 300.588215] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.588489] ? __pfx_ext4_create+0x10/0x10 [ 300.588704] lookup_open.isra.0+0x117c/0x1580 [ 300.588942] ? __pfx_lookup_open.isra.0+0x10/0x10 [ 300.589192] ? lock_acquire+0x80/0xb0 [ 300.589385] ? path_openat+0xd40/0x2bf0 [ 300.589597] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.589876] path_openat+0xdbf/0x2bf0 [ 300.590080] ? __pfx_path_openat+0x10/0x10 [ 300.590294] ? __this_cpu_preempt_check+0x21/0x30 [ 300.590540] ? lock_is_held_type+0xef/0x150 [ 300.590777] ? __this_cpu_preempt_check+0x21/0x30 [ 300.591026] do_filp_open+0x1cc/0x420 [ 300.591221] ? __pfx_do_filp_open+0x10/0x10 [ 300.591440] ? lock_release+0x441/0x870 [ 300.591652] ? do_raw_spin_unlock+0x15c/0x210 [ 300.591895] do_sys_openat2+0x185/0x1f0 [ 300.592097] ? __pfx_do_sys_openat2+0x10/0x10 [ 300.592327] ? __this_cpu_preempt_check+0x21/0x30 [ 300.592570] ? seqcount_lockdep_reader_access.constprop.0+0xb4/0xd0 [ 300.592891] ? lockdep_hardirqs_on+0x89/0x110 [ 300.593122] __x64_sys_openat+0x17a/0x240 [ 300.593332] ? __pfx___x64_sys_openat+0x10/0x10 [ 300.593569] ? __audit_syscall_entry+0x39c/0x500 [ 300.593816] x64_sys_call+0x1a52/0x20d0 [ 300.594017] do_syscall_64+0x6d/0x140 [ 300.594212] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 300.594470] RIP: 0033:0x7f4af113e8c4 [ 300.594657] RSP: 002b:00007f4af01fea40 EFLAGS: 00000293 ORIG_RAX: 0000000000000101 [ 300.595043] RAX: ffffffffffffffda RBX: 00007f4ae8000c50 RCX: 00007f4af113e8c4 [ 300.595396] RDX: 0000000000000241 RSI: 00007f4af01feb80 RDI: 00000000ffffff9c [ 300.595748] RBP: 00007f4af01feb80 R08: 0000000000000000 R09: 0000000000000001 [ 300.596099] R10: 00000000000001b6 R11: 0000000000000293 R12: 0000000000000241 [ 300.596451] R13: 00007f4ae8000c50 R14: 0000000000000001 R15: 00007f4ae8010540 [ 300.596818] </TASK> [ 300.596937] INFO: task rs:main Q:Reg:322 blocked for more than 147 seconds. [ 300.597279] Not tainted 6.12.0-rc4-kvm #1 [ 300.597508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.597889] task:rs:main Q:Reg state:D stack:0 pid:322 tgid:300 ppid:1 flags:0x00000002 [ 300.598345] Call Trace: [ 300.598479] <TASK> [ 300.598597] __schedule+0xe13/0x33a0 [ 300.598804] ? __pfx___schedule+0x10/0x10 [ 300.599013] ? lock_release+0x441/0x870 [ 300.599217] ? __pfx_lock_release+0x10/0x10 [ 300.599433] ? trace_lock_acquire+0x139/0x1b0 [ 300.599664] ? lock_acquire+0x80/0xb0 [ 300.599857] ? schedule+0x216/0x3f0 [ 300.600046] schedule+0xf6/0x3f0 [ 300.600221] io_schedule+0xce/0x150 [ 300.600408] bit_wait_io+0x1f/0x100 [ 300.600595] __wait_on_bit+0x68/0x1b0 [ 300.600791] ? __pfx_bit_wait_io+0x10/0x10 [ 300.601010] out_of_line_wait_on_bit+0xe6/0x120 [ 300.601246] ? __pfx_out_of_line_wait_on_bit+0x10/0x10 [ 300.601512] ? __pfx_wake_bit_function+0x10/0x10 [ 300.601753] ? __might_sleep+0xa3/0x170 [ 300.601959] do_get_write_access+0x83b/0x1010 [ 300.602196] jbd2_journal_get_write_access+0x1e4/0x270 [ 300.602462] __ext4_journal_get_write_access+0x72/0x3b0 [ 300.602744] ext4_reserve_inode_write+0x199/0x280 [ 300.602990] ? ext4_dirty_inode+0xeb/0x140 [ 300.603204] __ext4_mark_inode_dirty+0x18b/0x8c0 [ 300.603446] ? debug_smp_processor_id+0x20/0x30 [ 300.603683] ? __pfx___ext4_mark_inode_dirty+0x10/0x10 [ 300.603945] ? trace_jbd2_handle_start+0x77/0x210 [ 300.604192] ? jbd2__journal_start+0xff/0x6b0 [ 300.604425] ? __ext4_journal_start_sb+0x1f8/0x600 [ 300.604672] ? ext4_dirty_inode+0xb3/0x140 [ 300.604892] ? __pfx_ext4_dirty_inode+0x10/0x10 [ 300.605127] ext4_dirty_inode+0xeb/0x140 [ 300.605332] ? rcu_is_watching+0x19/0xc0 [ 300.605539] __mark_inode_dirty+0x1d0/0xd50 [ 300.605756] ? __pfx_generic_write_checks+0x10/0x10 [ 300.606011] generic_update_time+0xe0/0x110 [ 300.606232] file_modified+0x1af/0x1e0 [ 300.606432] ext4_buffered_write_iter+0x101/0x3e0 [ 300.606678] ext4_file_write_iter+0x8b8/0x1ab0 [ 300.606924] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 300.607175] ? debug_smp_processor_id+0x20/0x30 [ 300.607414] ? __pfx_ext4_file_write_iter+0x10/0x10 [ 300.607667] ? __this_cpu_preempt_check+0x21/0x30 [ 300.607912] ? lock_is_held_type+0xef/0x150 [ 300.608135] vfs_write+0xc46/0x1120 [ 300.608324] ? __pfx_ext4_file_write_iter+0x10/0x10 [ 300.608577] ? __pfx_vfs_write+0x10/0x10 [ 300.608797] ? mutex_lock_nested+0x1f/0x30 [ 300.609016] ksys_write+0x14f/0x290 [ 300.609206] ? __pfx_ksys_write+0x10/0x10 [ 300.609420] ? __audit_syscall_entry+0x39c/0x500 [ 300.609663] __x64_sys_write+0x7b/0xc0 [ 300.609860] ? syscall_trace_enter+0x14a/0x230 [ 300.610093] x64_sys_call+0x1685/0x20d0 [ 300.610294] do_syscall_64+0x6d/0x140 [ 300.610490] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 300.610763] RIP: 0033:0x7f4af113ebcf [ 300.610950] RSP: 002b:00007f4aefbff890 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 [ 300.611326] RAX: ffffffffffffffda RBX: 00007f4ae0000e70 RCX: 00007f4af113ebcf [ 300.611678] RDX: 0000000000000040 RSI: 00007f4ae0001150 RDI: 0000000000000005 [ 300.612031] RBP: 00007f4ae0001150 R08: 0000000000000000 R09: 0000000000000000 [ 300.612384] R10: 0000000000000007 R11: 0000000000000293 R12: 0000000000000000 [ 300.612738] R13: 000055979b901330 R14: 00007f4ae0000e70 R15: 0000000000000040 [ 300.613110] </TASK> [ 300.613251] INFO: task repro:684 blocked for more than 147 seconds. [ 300.613537] Not tainted 6.12.0-rc4-kvm #1 [ 300.613749] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 300.614135] task:repro state:D stack:0 pid:684 tgid:684 ppid:679 flags:0x00000006 [ 300.614606] Call Trace: [ 300.614754] <TASK> [ 300.614874] __schedule+0xe13/0x33a0 [ 300.615073] ? __pfx___schedule+0x10/0x10 [ 300.615283] ? lock_release+0x441/0x870 [ 300.615489] ? __pfx_lock_release+0x10/0x10 [ 300.615708] ? trace_lock_acquire+0x139/0x1b0 [ 300.615942] ? lock_acquire+0x80/0xb0 [ 300.616136] ? schedule+0x216/0x3f0 [ 300.616327] schedule+0xf6/0x3f0 [ 300.616507] io_schedule+0xce/0x150 [ 300.616695] blk_mq_get_tag+0x5e7/0xce0 [ 300.616909] ? __pfx_blk_mq_get_tag+0x10/0x10 [ 300.617146] ? __pfx_autoremove_wake_function+0x10/0x10 [ 300.617429] __blk_mq_alloc_requests+0x6f6/0x1720 [ 300.617678] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30 [ 300.617970] ? __pfx___blk_mq_alloc_requests+0x10/0x10 [ 300.618236] ? __this_cpu_preempt_check+0x21/0x30 [ 300.618480] ? lock_acquire.part.0+0x152/0x390 [ 300.618733] blk_mq_alloc_request+0x655/0x930 [ 300.618967] ? __pfx_blk_mq_alloc_request+0x10/0x10 [ 300.619234] scsi_execute_cmd+0x223/0x1040 [ 300.619455] ? debug_smp_processor_id+0x20/0x30 [ 300.619695] ? __pfx_scsi_execute_cmd+0x10/0x10 [ 300.619931] ? rcu_is_watching+0x19/0xc0 [ 300.620139] ? __kmalloc_noprof+0x34f/0x4a0 [ 300.620358] ? ata_cmd_ioctl+0x375/0x570 [ 300.620570] ata_cmd_ioctl+0x23c/0x570 [ 300.620775] ? __pfx_ata_cmd_ioctl+0x10/0x10 [ 300.621000] ? lock_release+0x441/0x870 [ 300.621209] ? __this_cpu_preempt_check+0x21/0x30 [ 300.621463] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.621740] ? security_capable+0x19d/0x1e0 [ 300.621976] ata_sas_scsi_ioctl+0x60a/0x7c0 [ 300.622195] ? mark_lock.part.0+0xf3/0x17b0 [ 300.622415] ? __pfx_ata_sas_scsi_ioctl+0x10/0x10 [ 300.622658] ? __pfx_mark_lock.part.0+0x10/0x10 [ 300.622915] ? __this_cpu_preempt_check+0x21/0x30 [ 300.623161] ? lock_is_held_type+0xef/0x150 [ 300.623382] ? __pfx_ata_scsi_ioctl+0x10/0x10 [ 300.623610] ata_scsi_ioctl+0x70/0x90 [ 300.623807] scsi_ioctl+0xb6f/0x1630 [ 300.624001] ? __might_sleep+0xa3/0x170 [ 300.624208] ? __pfx_scsi_ioctl+0x10/0x10 [ 300.624419] ? scsi_block_when_processing_errors+0x2c4/0x390 [ 300.624711] ? __pfx_scsi_block_when_processing_errors+0x10/0x10 [ 300.625022] ? __pfx___lock_acquire+0x10/0x10 [ 300.625250] ? __sanitizer_cov_trace_switch+0x58/0xa0 [ 300.625512] ? __sanitizer_cov_trace_const_cmp2+0x1c/0x30 [ 300.625787] ? do_vfs_ioctl+0x4fa/0x1920 [ 300.625997] ? __pfx_do_vfs_ioctl+0x10/0x10 [ 300.626217] ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 [ 300.626497] sg_ioctl+0xfb8/0x2880 [ 300.626686] ? __this_cpu_preempt_check+0x21/0x30 [ 300.626947] ? lock_release+0x441/0x870 [ 300.627155] ? __pfx_sg_ioctl+0x10/0x10 [ 300.627366] ? __this_cpu_preempt_check+0x21/0x30 [ 300.627615] ? seqcount_lockdep_reader_access.constprop.0+0xb4/0xd0 [ 300.627932] ? lockdep_hardirqs_on+0x89/0x110 [ 300.628160] ? trace_hardirqs_on+0x51/0x60 [ 300.628381] ? seqcount_lockdep_reader_access.constprop.0+0xc0/0xd0 [ 300.628697] ? __sanitizer_cov_trace_cmp4+0x1a/0x20 [ 300.628949] ? ktime_get_coarse_real_ts64+0xbf/0xf0 [ 300.629210] ? __pfx_sg_ioctl+0x10/0x10 [ 300.629421] __x64_sys_ioctl+0x1b5/0x230 [ 300.629639] x64_sys_call+0x1209/0x20d0 [ 300.629844] do_syscall_64+0x6d/0x140 [ 300.630044] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 300.630310] RIP: 0033:0x7fc80ee3ee5d [ 300.630502] RSP: 002b:00007fffa979c518 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 [ 300.630915] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fc80ee3ee5d [ 300.631281] RDX: 0000000020000080 RSI: 000000000000031f RDI: 0000000000000003 [ 300.631640] RBP: 00007fffa979c530 R08: 00007fffa979c2a8 R09: 00007fffa979c530 [ 300.631997] R10: 0000000000000000 R11: 0000000000000206 R12: 00007fffa979c688 [ 300.632354] R13: 0000000000401972 R14: 0000000000403e08 R15: 00007fc80f193000 [ 300.632722] </TASK> [ 300.632849] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [ 300.633253] [ 300.633253] Showing all locks held in the system: [ 300.633567] 7 locks held by kworker/u8:0/11: [ 300.633790] #0: ffff88800e4a8948 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x11fc/0x1b50 [ 300.634296] #1: ffff88800dcd7d78 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x8a0/0x1b0 [ 300.634871] #2: ffff888011f480e0 (&type->s_umount_key#38){++++}-{3:3}, at: super_trylock_shared+0x2b/0x110 [ 300.635370] #3: ffff888011f4ab98 (&sbi->s_writepages_rwsem){++++}-{0:0}, at: do_writepages+0x1c3/0x8a0 [ 300.635848] #4: ffff888011f4c958 (jbd2_handle){++++}-{0:0}, at: start_this_handle+0x107b/0x1590 [ 300.636299] #5: ffff888023bb33c0 (&ei->i_data_sem){++++}-{3:3}, at: ext4_map_blocks+0x344/0x1390 [ 300.636754] #6: ffffe8ffffc077f8 (&lg->lg_mutex){+.+.}-{3:3}, at: ext4_mb_initialize_context+0x8a3/0xd20 [ 300.637257] 4 locks held by kworker/u8:1/31: [ 300.637479] #0: ffff88800e4a8948 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x11fc/0x1b50 [ 300.637979] #1: ffff88800df47d78 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x8a0/0x1b0 [ 300.638532] #2: ffff888011f480e0 (&type->s_umount_key#38){++++}-{3:3}, at: super_trylock_shared+0x2b/0x110 [ 300.639046] #3: ffff888011f4ab98 (&sbi->s_writepages_rwsem){++++}-{0:0}, at: do_writepages+0x1c3/0x8a0 [ 300.639525] 1 lock held by khungtaskd/33: [ 300.639732] #0: ffffffff8705ca00 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x73/0x3c0 [ 300.640191] 4 locks held by kworker/u8:2/35: [ 300.640411] #0: ffff88800e4a8948 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x11fc/0x1b50 [ 300.640917] #1: ffff88800e4bfd78 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x8a0/0x1b0 [ 300.641479] #2: ffff888011f480e0 (&type->s_umount_key#38){++++}-{3:3}, at: super_trylock_shared+0x2b/0x110 [ 300.641990] #3: ffff888011f4ab98 (&sbi->s_writepages_rwsem){++++}-{0:0}, at: do_writepages+0x1c3/0x8a0 [ 300.642483] 3 locks held by kworker/u8:4/167: [ 300.642709] #0: ffff88800e4a8948 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x11fc/0x1b50 [ 300.643228] #1: ffff888013e07d78 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x8a0/0x1b0 [ 300.643782] #2: ffff88800d77e0e0 (&type->s_umount_key#52){.+.+}-{3:3}, at: super_trylock_shared+0x2b/0x110 [ 300.644282] 4 locks held by kworker/1:3/233: [ 300.644503] #0: ffff88800d45ad48 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x11fc/0x1b50 [ 300.644995] #1: ffff888014ea7d78 ((work_completion)(&(&ap->scsi_rescan_task)->work)){+.+.}-{0:0}, at: process_one_work0 [ 300.645610] #2: ffff88801042c430 (&ap->scsi_scan_mutex){+.+.}-{3:3}, at: ata_scsi_dev_rescan+0x42/0x470 [ 300.646096] #3: ffff8880103e8378 (&dev->mutex){....}-{3:3}, at: scsi_rescan_device+0x33/0x360 [ 300.646542] 4 locks held by auditd/240: [ 300.646757] #0: ffff8880111580f8 (&f->f_pos_lock){+.+.}-{3:3}, at: fdget_pos+0x25e/0x380 [ 300.647182] #1: ffff888011f48408 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x14f/0x290 [ 300.647605] #2: ffff888017c8d2b8 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xb8/0x3e0 [ 300.648137] #3: ffff888011f4c958 (jbd2_handle){++++}-{0:0}, at: start_this_handle+0x107b/0x1590 [ 300.648591] 3 locks held by in:imjournal/319: [ 300.648820] #0: ffff888011f48408 (sb_writers#3){.+.+}-{0:0}, at: path_openat+0x1f7c/0x2bf0 [ 300.649260] #1: ffff88801ae1f040 (&type->i_mutex_dir_key#3){++++}-{3:3}, at: path_openat+0xd40/0x2bf0 [ 300.649745] #2: ffff888011f4c958 (jbd2_handle){++++}-{0:0}, at: start_this_handle+0x107b/0x1590 [ 300.650205] 4 locks held by rs:main Q:Reg/322: [ 300.650437] #0: ffff888013b782b8 (&f->f_pos_lock){+.+.}-{3:3}, at: fdget_pos+0x25e/0x380 [ 300.650880] #1: ffff888011f48408 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x14f/0x290 [ 300.651306] #2: ffff8880195183f8 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xb8/0x3e0 [ 300.651844] #3: ffff888011f4c958 (jbd2_handle){++++}-{0:0}, at: start_this_handle+0x107b/0x1590 [ 300.652307] [ 300.652397] ============================================= > Kind regards, > Niklas
