On Wed, Jan 06, 2021 at 11:24:28AM +0100, Jan Kara wrote:
> Currently when non-mq aware IO scheduler (BFQ, mq-deadline) is used for
> a queue with multiple HW queues, the performance it rather bad. The
> problem is that these IO schedulers use queue-wide locking and their
> dispatch function does not respect the hctx it is passed in and returns
> any request it finds appropriate. Thus locality of request access is
> broken and dispatch from multiple CPUs just contends on IO scheduler
> locks. For these IO schedulers there's little point in dispatching from
> multiple CPUs. Instead dispatch always only from a single CPU to limit
> contention.
>
> Below is a comparison of dbench runs on XFS filesystem where the storage
> is a raid card with 64 HW queues and to it attached a single rotating
> disk. BFQ is used as IO scheduler:
>
> clients MQ SQ MQ-Patched
> Amean 1 39.12 (0.00%) 43.29 * -10.67%* 36.09 * 7.74%*
> Amean 2 128.58 (0.00%) 101.30 * 21.22%* 96.14 * 25.23%*
> Amean 4 577.42 (0.00%) 494.47 * 14.37%* 508.49 * 11.94%*
> Amean 8 610.95 (0.00%) 363.86 * 40.44%* 362.12 * 40.73%*
> Amean 16 391.78 (0.00%) 261.49 * 33.25%* 282.94 * 27.78%*
> Amean 32 324.64 (0.00%) 267.71 * 17.54%* 233.00 * 28.23%*
> Amean 64 295.04 (0.00%) 253.02 * 14.24%* 242.37 * 17.85%*
> Amean 512 10281.61 (0.00%) 10211.16 * 0.69%* 10447.53 * -1.61%*
>
> Numbers are times so lower is better. MQ is stock 5.10-rc6 kernel. SQ is
> the same kernel with megaraid_sas.host_tagset_enable=0 so that the card
> advertises just a single HW queue. MQ-Patched is a kernel with this
> patch applied.
>
> You can see multiple hardware queues heavily hurt performance in
> combination with BFQ. The patch restores the performance.
>
> Signed-off-by: Jan Kara <jack@xxxxxxx>
> ---
> block/blk-mq.c | 62 ++++++++++++++++++++++++++++++++++------
> block/kyber-iosched.c | 1 +
> include/linux/elevator.h | 2 ++
> 3 files changed, 56 insertions(+), 9 deletions(-)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 57f3482b2c26..26e0f6e64a3a 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -63,15 +63,20 @@ static int blk_mq_poll_stats_bkt(const struct request *rq)
> return bucket;
> }
>
> +/* Check if there are requests queued in hctx lists. */
> +static bool blk_mq_hctx_has_queued_rq(struct blk_mq_hw_ctx *hctx)
> +{
> + return !list_empty_careful(&hctx->dispatch) ||
> + sbitmap_any_bit_set(&hctx->ctx_map);
> +}
> +
blk_mq_hctx_mark_pending() is only called in case of none scheduler, so
looks not necessary to check hctx->ctx_map in blk_mq_hctx_has_queued_rq()
which is supposed to be used when real io scheduler is attached to MQ queue.
Thanks,
Ming
