Hi, an issue that might have to do with this one, reported for a ufs platform in scsi LKML - https://www.spinics.net/lists/linux-scsi/msg126868.html. There, they suspected that blk_mq_requeue_work fires while the platform is in system suspend. In ufs, system suspend sends SSU to the device, and just another SSU wakes it up. Can you trace your platform if this is happening while the device has suspended? Thanks, Avri From: Zak Hays <mailto:zak.hays@xxxxxxxxxxx> Date: Fri, Jan 11, 2019 at 12:05 AM Subject: mmc hung tasks at boot To: mailto:linux-mmc@xxxxxxxxxxxxxxx <mailto:linux-mmc@xxxxxxxxxxxxxxx> Cc: Bradley Bolen <mailto:bradley.bolen@xxxxxxxxxxx>, Steven Walter <mailto:steven.walter@xxxxxxxxxxx> Hello all, After upgrading to kernel version v4.17, I see hangs one out of every 200 boots or so. I then see the following hung tasks: INFO: task kblockd:30 blocked for more than 120 seconds. Tainted: P O 4.17.19-yocto-standard-edf324cbd3b997d05686954a2e8e5d27 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kblockd D 0 30 2 0x00000000 Workqueue: kblockd blk_mq_run_work_fn [<c064382c>] (__schedule) from [<c0643a6c>] (schedule+0xa4/0xd0) [<c0643a6c>] (schedule) from [<c04ffac0>] (__mmc_claim_host+0x12c/0x238) [<c04ffac0>] (__mmc_claim_host) from [<c04ffc04>] (mmc_get_card+0x38/0x3c) [<c04ffc04>] (mmc_get_card) from [<c0513e44>] (mmc_mq_queue_rq+0x104/0x1fc) [<c0513e44>] (mmc_mq_queue_rq) from [<c02f8378>] (blk_mq_dispatch_rq_list+0x380/0x4b0) [<c02f8378>] (blk_mq_dispatch_rq_list) from [<c02fc2cc>] (blk_mq_do_dispatch_sched+0xf8/0x110) [<c02fc2cc>] (blk_mq_do_dispatch_sched) from [<c02fca38>] (blk_mq_sched_dispatch_requests+0x160/0x1d0) [<c02fca38>] (blk_mq_sched_dispatch_requests) from [<c02f63b4>] (__blk_mq_run_hw_queue+0x120/0x168) [<c02f63b4>] (__blk_mq_run_hw_queue) from [<c02f6434>] (blk_mq_run_work_fn+0x38/0x3c) [<c02f6434>] (blk_mq_run_work_fn) from [<c0047890>] (process_one_work+0x288/0x474) [<c0047890>] (process_one_work) from [<c0047ab4>] (process_scheduled_works+0x38/0x3c) [<c0047ab4>] (process_scheduled_works) from [<c00486a8>] (rescuer_thread+0x1f8/0x35c) [<c00486a8>] (rescuer_thread) from [<c004d948>] (kthread+0x158/0x174) [<c004d948>] (kthread) from [<c00090e4>] (ret_from_fork+0x14/0x30) Exception stack(0xe1dd3fb0 to 0xe1dd3ff8) 3fa0: 00000000 00000000 00000000 00000000 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 INFO: task kworker/1:1H:91 blocked for more than 120 seconds. Tainted: P O 4.17.19-yocto-standard-edf324cbd3b997d05686954a2e8e5d27 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/1:1H D 0 91 2 0x00000000 Workqueue: kblockd blk_mq_run_work_fn [<c064382c>] (__schedule) from [<c0643a6c>] (schedule+0xa4/0xd0) [<c0643a6c>] (schedule) from [<c04ffac0>] (__mmc_claim_host+0x12c/0x238) [<c04ffac0>] (__mmc_claim_host) from [<c04ffc04>] (mmc_get_card+0x38/0x3c) [<c04ffc04>] (mmc_get_card) from [<c0513e44>] (mmc_mq_queue_rq+0x104/0x1fc) [<c0513e44>] (mmc_mq_queue_rq) from [<c02f8378>] (blk_mq_dispatch_rq_list+0x380/0x4b0) [<c02f8378>] (blk_mq_dispatch_rq_list) from [<c02fc2cc>] (blk_mq_do_dispatch_sched+0xf8/0x110) [<c02fc2cc>] (blk_mq_do_dispatch_sched) from [<c02fca38>] (blk_mq_sched_dispatch_requests+0x160/0x1d0) [<c02fca38>] (blk_mq_sched_dispatch_requests) from [<c02f63b4>] (__blk_mq_run_hw_queue+0x120/0x168) [<c02f63b4>] (__blk_mq_run_hw_queue) from [<c02f6434>] (blk_mq_run_work_fn+0x38/0x3c) [<c02f6434>] (blk_mq_run_work_fn) from [<c0047890>] (process_one_work+0x288/0x474) [<c0047890>] (process_one_work) from [<c0048abc>] (worker_thread+0x2b0/0x428) [<c0048abc>] (worker_thread) from [<c004d948>] (kthread+0x158/0x174) [<c004d948>] (kthread) from [<c00090e4>] (ret_from_fork+0x14/0x30) Exception stack(0xc19cffb0 to 0xc19cfff8) ffa0: 00000000 00000000 00000000 00000000 ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 After bisecting through the commits, I've found the hangs started after this commit: 81196976ed94 Adrian Hunter Wed Nov 29 15:41:03 2017 +0200 mmc: block: Add blk-mq support I'm not sure however what about this particular commit is the source of the problem. It appears like multiple tasks are trying to claim the host but whatever task is responsible for releasing it isn't getting triggered. If I dump the blocked tasks, I don't see any other mmc-related tasks other than the two above. Has anyone run into this issue before? If not, does anyone have any ideas what might be causing the problem? Thanks, Zak Hays -- http://www.sigbee.com/?utm_campaign=signature&origUserUID=8f599119d5579f88613d9ed76b4e4be089057c3e&origSigUserImage=5834cb998768feb9.gif ;
