This patch has now been tested in the customer environment and results
were good (fixed the hangs).
On Mon, Feb 7, 2022 at 9:45 PM Ming Lei <ming.lei@xxxxxxxxxx> wrote:
>
> On Tue, Feb 1, 2022 at 4:34 AM David Jeffery <djeffery@xxxxxxxxxx> wrote:
> >
> > When blk_mq_delay_run_hw_queues sets an hctx to run in the future, it can
> > reset the delay length for an already pending delayed work run_work. This
> > creates a scenario where multiple hctx may have their queues set to run,
> > but if one runs first and finds nothing to do, it can reset the delay of
> > another hctx and stall the other hctx's ability to run requests.
> >
> > To avoid this I/O stall when an hctx's run_work is already pending,
> > leave it untouched to run at its current designated time rather than
> > extending its delay. The work will still run which keeps closed the race
> > calling blk_mq_delay_run_hw_queues is needed for while also avoiding the
> > I/O stall.
> >
> > Signed-off-by: David Jeffery <djeffery@xxxxxxxxxx>
> > ---
> > block/blk-mq.c | 8 ++++++++
> > 1 file changed, 8 insertions(+)
> >
> >
> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index f3bf3358a3bb..ae46eb4bf547 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -2177,6 +2177,14 @@ void blk_mq_delay_run_hw_queues(struct request_queue *q, unsigned long msecs)
> > queue_for_each_hw_ctx(q, hctx, i) {
> > if (blk_mq_hctx_stopped(hctx))
> > continue;
> > + /*
> > + * If there is already a run_work pending, leave the
> > + * pending delay untouched. Otherwise, a hctx can stall
> > + * if another hctx is re-delaying the other's work
> > + * before the work executes.
> > + */
> > + if (delayed_work_pending(&hctx->run_work))
> > + continue;
>
> The issue is triggered on BFQ, since BFQ's has_work() may return true,
> however its ->dispatch_request() may return NULL, so
> blk_mq_delay_run_hw_queues()
> is run for delay schedule.
>
> In case of multiple hw queue, the described issue may be triggered, and cause io
> stall for long time. And there are only 3 in-tree callers of
> blk_mq_delay_run_hw_queues(),
> David's fix works well for the 3 users, so this patch looks fine:
>
> Reviewed-by: Ming Lei <ming.lei@xxxxxxxxxx>
>
> Thanks,
>
