This is a note to let you know that I've just added the patch titled
block: do not reverse request order when flushing plug list
to the 6.1-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
block-do-not-reverse-request-order-when-flushing-plu.patch
and it can be found in the queue-6.1 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@xxxxxxxxxxxxxxx> know about it.
commit e9280b98420d2bf4c97dde84058d362610beb048
Author: Jan Kara <jack@xxxxxxx>
Date: Mon Mar 13 10:30:02 2023 +0100
block: do not reverse request order when flushing plug list
[ Upstream commit 34e0a279a993debaff03158fc2fbf6a00c093643 ]
Commit 26fed4ac4eab ("block: flush plug based on hardware and software
queue order") changed flushing of plug list to submit requests one
device at a time. However while doing that it also started using
list_add_tail() instead of list_add() used previously thus effectively
submitting requests in reverse order. Also when forming a rq_list with
remaining requests (in case two or more devices are used), we
effectively reverse the ordering of the plug list for each device we
process. Submitting requests in reverse order has negative impact on
performance for rotational disks (when BFQ is not in use). We observe
10-25% regression in random 4k write throughput, as well as ~20%
regression in MariaDB OLTP benchmark on rotational storage on btrfs
filesystem.
Fix the problem by preserving ordering of the plug list when inserting
requests into the queuelist as well as by appending to requeue_list
instead of prepending to it.
Fixes: 26fed4ac4eab ("block: flush plug based on hardware and software queue order")
Signed-off-by: Jan Kara <jack@xxxxxxx>
Reviewed-by: Christoph Hellwig <hch@xxxxxx>
Link: https://lore.kernel.org/r/20230313093002.11756-1-jack@xxxxxxx
Signed-off-by: Jens Axboe <axboe@xxxxxxxxx>
Signed-off-by: Sasha Levin <sashal@xxxxxxxxxx>
diff --git a/block/blk-mq.c b/block/blk-mq.c
index fe0a3a882f465..aa67a52c5a069 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2711,6 +2711,7 @@ static void blk_mq_dispatch_plug_list(struct blk_plug *plug, bool from_sched)
struct blk_mq_hw_ctx *this_hctx = NULL;
struct blk_mq_ctx *this_ctx = NULL;
struct request *requeue_list = NULL;
+ struct request **requeue_lastp = &requeue_list;
unsigned int depth = 0;
LIST_HEAD(list);
@@ -2721,10 +2722,10 @@ static void blk_mq_dispatch_plug_list(struct blk_plug *plug, bool from_sched)
this_hctx = rq->mq_hctx;
this_ctx = rq->mq_ctx;
} else if (this_hctx != rq->mq_hctx || this_ctx != rq->mq_ctx) {
- rq_list_add(&requeue_list, rq);
+ rq_list_add_tail(&requeue_lastp, rq);
continue;
}
- list_add_tail(&rq->queuelist, &list);
+ list_add(&rq->queuelist, &list);
depth++;
} while (!rq_list_empty(plug->mq_list));
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index d6119c5d1069b..a9764cbf7f8d2 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -228,6 +228,12 @@ static inline unsigned short req_get_ioprio(struct request *req)
*(listptr) = rq; \
} while (0)
+#define rq_list_add_tail(lastpptr, rq) do { \
+ (rq)->rq_next = NULL; \
+ **(lastpptr) = rq; \
+ *(lastpptr) = &rq->rq_next; \
+} while (0)
+
#define rq_list_pop(listptr) \
({ \
struct request *__req = NULL; \
