Hi, On Sun, Jul 23, 2023 at 10:28 PM Sherry Yang <sherry.yang@xxxxxxxxxx> wrote: > > Hi Douglas, > > We observed linux-stable v5.4 VM boot hangs, but probably only 1 in thousands of boots (less than 10,000 boots). We started 16 VMs on a Bare Metal with loop reboots, I chose 10,000 boots as a threshold, and bisected it. After a painful bisection, I found the culprit commit 578c8f09c04b (“Revert scsi: core: run queue if SCSI device queue isnt ready and queue is idle”). This commit actually was merged to v5.8 the 1st time. It's a series of patch set (https://www.spinics.net/lists/linux-block/msg51866.html). Actually, in the 4-patch series, 2 of them have already been backported to linux-stable v5.4, but not at the same time: > > 1) ab3cee3762e5 (“blk-mq: In blk_mq_dispatch_rq_list() no budget is a reason to kick”) in tag v5.4.86 > 2) 578c8f09c04b (“Revert scsi: core: run queue if SCSI device queue isnt ready and queue is idle”) in tag v5.4.235, it’s backported as stable dependency for another commit > > Signed-off-by: Douglas Anderson <dianders@xxxxxxxxxxxx> > Reviewed-by: Ming Lei <ming.lei@xxxxxxxxxx> > Acked-by: Martin K. Petersen <martin.petersen@xxxxxxxxxx> > Signed-off-by: Jens Axboe <axboe@xxxxxxxxx> > Stable-dep-of: c31e76bcc379 ("blk-mq: remove stale comment for blk_mq_sched_mark_restart_hctx”) > Signed-off-by: Sasha Levin <sashal@xxxxxxxxxx> > > And I tried backporting the other 2 patches to v5.4, the issue is still reproducible. > > I tested multiple kernels, the issue is not reproducible within 10,000 boots in the following kernels: > 1) Linux v5.9 > 2) Linux v5.4.249 + revert of 578c8f09c04b (“Revert scsi: core: run queue if SCSI device queue isnt ready and queue is idle”) > > Not exactly sure how this commit is affecting linux-stable v5.4, but I suspect some prerequisite commits are missing which lead to boot hangs on linux-stable v5.4 but not on higher releases. Could you take a look at this issue and share your insight? Ugh, I spent many days pouring over the code and digging through debug traces in order to write those patches. I don't think I'd be able to give any concrete advice without spending many days and being able to reproduce multiple times with traces since pretty much any knowledge I learned during the course of developing those patches has decayed over the last several years. :( I don't happen to know any dependencies offhand... That being said, it seems like: 1. Backporting the revert (the 4th patch in the series) without all the other patches in the series feels wrong. In the text of the revert I explicitly refer to the other patches in the series as prerequisites. I guess you said you tried backporting the other two patches and they didn't help, though? That's no good. :( 2. I don't think the revert is actually important to backport to stable. While the first 3 patches were important to fix the problems I was seeing, the revert was just a cleanup. If the revert is causing problems in 5.4.x then I'd suggest removing it from 5.4.x Does that make sense? So ideally you'd submit 3 patches to the stable kernel: a) Revert the revert b) Pick ("blk-mq: Add blk_mq_delay_run_hw_queues() API call") c) Pick ("blk-mq: Rerun dispatching in the case of budget contention") FWIW, we seem to have all 4 patches in the ChromeOS 5.4 kernel tree. They all landed together plus 1 prerequisite. * https://crrev.com/c/2155423 - FROMGIT: Revert "scsi: core: run queue if SCSI device queue isn't ready and queue is idle" * https://crrev.com/c/2133069 - FROMGIT: blk-mq: Rerun dispatching in the case of budget contention * https://crrev.com/c/2155422 - FROMGIT: blk-mq: Add blk_mq_delay_run_hw_queues() API call * https://crrev.com/c/2125232 - FROMGIT: blk-mq: In blk_mq_dispatch_rq_list() "no budget" is a reason to kick * https://crrev.com/c/2155421 - UPSTREAM: blk-mq: Put driver tag in blk_mq_dispatch_rq_list() when no budget When I saw the prerequisite in there I was hopeful that it was the one you needed, but it looks like that's already in 5.4 stable so (I presume) you've already been testing with it... -Doug
