On Tue, Feb 24 2015 at 1:32pm -0500,
Mike Snitzer <snitzer@xxxxxxxxxx> wrote:
On Tue, Feb 24 2015 at 1:16pm -0500,
Jens Axboe <axboe@xxxxxxxxx> wrote:
So all of this needs to be tested and performance vetted. But my
original suggestion was something like:
if (run_queue && !md->queue->nr_pending)
blk_run_queue_async(md->queue);
which might be a bit extreme, but if we hit 0, that's the only case
where you truly do need to run the queue. So that kind of logic
would give you the highest chance of merge success, potentially at
the cost of reduced performance for other cases.
Yeah, I was wondering about not running the queue at all when discussing
with Jeff earlier today. Seemed extreme, and Jeff thought it could
cause performance to really take a hit.
Whether you have to or not depends on how you break out of queueing
loops. But you definitely don't have to run it on every single
request completion...
OK, thanks for clarifying.
Will see how the initial RFC patch I shared works for Netapp's testcase
but based on those results will work to arrive at a more
generic/intelligent solution.
That initial RFC didn't do well. And given my results below, I'm not
holding out any hope for this io completion change, which we already
discussed as being more sane, having any impact:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=9068f2f15d5fc7a5c9dc6e6963913216ede0c745
Here is the sorry state of affairs:
Summary:
========
Request-based DM is somehow breaking the block layer's ability to properly merge requests.
Only good news is I seem to have a testbed that I can use to chase this issue down.
Test kernel is v4.0-rc1 with a few DM patches, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=for-next
baseline:
=========
Writing to an STEC PCIe SSD using 64 threads, each using 4K sequentional IOs.
There are clearly merges happening.
fio job:
--------
[64_seq_write]
filename=/dev/skd0
rw=write
rwmixread=0
blocksize=4k
iodepth=16
direct=1
numjobs=64
#nrfiles=1
runtime=100
ioengine=libaio
time_based
deadline:
---------
Run status group 0 (all jobs):
WRITE: io=33356MB, aggrb=341513KB/s, minb=4082KB/s, maxb=5463KB/s, mint=100001msec, maxt=100014msec
Disk stats (read/write):
skd0: ios=0/638099, merge=0/7483318, ticks=0/8015451, in_queue=8073672, util=100.00%
cfq:
----
rotational=0
------------
Run status group 0 (all jobs):
WRITE: io=20970MB, aggrb=214663KB/s, minb=3202KB/s, maxb=3620KB/s, mint=100001msec, maxt=100030msec
Disk stats (read/write):
skd0: ios=47/390277, merge=0/4939075, ticks=0/7082618, in_queue=7109995, util=100.00%
rotational=1
------------
Run status group 0 (all jobs):
WRITE: io=30075MB, aggrb=307130KB/s, minb=2613KB/s, maxb=10247KB/s, mint=100001msec, maxt=100271msec
Disk stats (read/write):
skd0: ios=57/2217835, merge=0/5414906, ticks=0/6110715, in_queue=6120362, util=99.97%
multipath:
==========
Same workload as above, but this time through a request-based DM multipath target.
There are clearly _no_ merges happening.
echo "0 1563037696 multipath 0 0 1 1 service-time 0 1 2 /dev/skd0 1000 1" | dmsetup create skd_mpath
[64_seq_write]
#filename=/dev/mapper/skd_mpath
filename=/dev/dm-9
rw=write
rwmixread=0
blocksize=4k
iodepth=16
direct=1
numjobs=64
#nrfiles=1
runtime=100
ioengine=libaio
time_based
deadline:
---------
Run status group 0 (all jobs):
WRITE: io=22559MB, aggrb=230976KB/s, minb=3565KB/s, maxb=3659KB/s, mint=100001msec, maxt=100011msec
Disk stats (read/write):
dm-9: ios=71/5772755, merge=0/0, ticks=3/15309410, in_queue=15360084, util=100.00%, aggrios=164/5775057, aggrmerge=0/0, aggrticks=16/14960469, aggrin_queue=14956943, aggrutil=99.92%
skd0: ios=164/5775057, merge=0/0, ticks=16/14960469, in_queue=14956943, util=99.92%
cfq:
----
rotational=0
------------
Run status group 0 (all jobs):
WRITE: io=19477MB, aggrb=199424KB/s, minb=3001KB/s, maxb=3268KB/s, mint=100001msec, maxt=100010msec
Disk stats (read/write):
dm-9: ios=40/4811814, merge=0/159762, ticks=0/14866276, in_queue=14912510, util=100.00%, aggrios=224/4814462, aggrmerge=0/0, aggrticks=13/14399688, aggrin_queue=14396342, aggrutil=99.92%
skd0: ios=224/4814462, merge=0/0, ticks=13/14399688, in_queue=14396342, util=99.92%
rotational=1
------------
Run status group 0 (all jobs):
WRITE: io=21051MB, aggrb=215188KB/s, minb=2147KB/s, maxb=8671KB/s, mint=100001msec, maxt=100175msec
Disk stats (read/write):
dm-9: ios=59/1468500, merge=0/3836597, ticks=2/5928782, in_queue=5959348, util=100.00%, aggrios=152/1469016, aggrmerge=0/0, aggrticks=5/395471, aggrin_queue=395287, aggrutil=71.17%
skd0: ios=152/1469016, merge=0/0, ticks=5/395471, in_queue=395287, util=71.17%
