Hi,
Tools like sar and iostat are reporting abnormally high %util with 4.19.y
running in VM (the disk is almost idle):
$ sar -dp
Linux 4.19.107-1.x86_64 03/25/20 _x86_64_ (6 CPU)
00:00:00 DEV tps ... %util
00:10:00 vda 0.55 ... 98.07
...
10:00:00 vda 0.44 ... 99.74
Average: vda 0.48 ... 98.98
The numbers look reasonable for the partition:
# iostat -x -p ALL 1 1
Linux 4.19.107-1.x86_64 03/25/20 _x86_64_ (6 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
10.51 0.00 8.58 0.05 0.11 80.75
Device r/s ... %util
vda 0.02 ... 98.25
vda1 0.01 ... 0.09
Lots of io_ticks in /proc/diskstats:
# cat /proc/uptime
45787.03 229321.29
# grep vda /proc/diskstats
253 0 vda 760 0 38498 731 28165 43212 1462928 157514 0 44690263
44812032 0 0 0 0
253 1 vda1 350 0 19074 293 26169 43212 1462912 154931 0 41560 150998
0 0 0 0
Other people are apparently seeing this too with 4.19:
https://kudzia.eu/b/2019/09/iostat-x-1-reporting-100-utilization-of-nearly-idle-nvme-drives/
I also see this only in 4.19.y and bisected to this (based on the Fixes
tag, this should have been taken to 4.14 too...):
commit 6131837b1de66116459ef4413e26fdbc70d066dc
Author: Omar Sandoval <osandov@xxxxxx>
Date: Thu Apr 26 00:21:58 2018 -0700
blk-mq: count allocated but not started requests in iostats inflight
In the legacy block case, we increment the counter right after we
allocate the request, not when the driver handles it. In both the legacy
and blk-mq cases, part_inc_in_flight() is called from
blk_account_io_start() right after we've allocated the request. blk-mq
only considers requests started requests as inflight, but this is
inconsistent with the legacy definition and the intention in the code.
This removes the started condition and instead counts all allocated
requests.
Fixes: f299b7c7a9de ("blk-mq: provide internal in-flight variant")
Signed-off-by: Omar Sandoval <osandov@xxxxxx>
Signed-off-by: Jens Axboe <axboe@xxxxxxxxx>
diff --git a/block/blk-mq.c b/block/blk-mq.c
index c3621453ad87..5450cbc61f8d 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -95,18 +95,15 @@ static void blk_mq_check_inflight(struct blk_mq_hw_ctx
*hctx,
{
struct mq_inflight *mi = priv;
- if (blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT) {
- /*
- * index[0] counts the specific partition that was asked
- * for. index[1] counts the ones that are active on the
- * whole device, so increment that if mi->part is indeed
- * a partition, and not a whole device.
- */
- if (rq->part == mi->part)
- mi->inflight[0]++;
- if (mi->part->partno)
- mi->inflight[1]++;
- }
+ /*
+ * index[0] counts the specific partition that was asked for.
index[1]
+ * counts the ones that are active on the whole device, so
increment
+ * that if mi->part is indeed a partition, and not a whole device.
+ */
+ if (rq->part == mi->part)
+ mi->inflight[0]++;
+ if (mi->part->partno)
+ mi->inflight[1]++;
}
void blk_mq_in_flight(struct request_queue *q, struct hd_struct *part,
If I get it right, when the disk is idle, and some request is allocated,
part_round_stats() with this commit will now add all ticks between
previous I/O and current time (now - part->stamp) to io_ticks.
Before the commit, part_round_stats() would only update part->stamp when
called after request allocation.
Any thoughts how to best fix this in 4.19?
I see the io_ticks accounting has been reworked in 5.0, do we need to
backport those to 4.19, or any ill effects if this commit is reverted in
4.19?
-Tommi
