On 01/27/2016 11:16 AM, Mike Snitzer wrote:
On Wed, Jan 27 2016 at 12:51pm -0500,
Jens Axboe <axboe@xxxxxxxxx> wrote:
On 01/27/2016 10:48 AM, Mike Snitzer wrote:
BTW, I _cannot_ get null_blk to come even close to your reported 1500K+
IOPs on 2 "fast" systems I have access to. Which arguments are you
loading the null_blk module with?
I've been using:
modprobe null_blk gb=4 bs=4096 nr_devices=1 queue_mode=2 submit_queues=12
On my 1 system is a 12 core single socket, single NUMA node with 12G of
memory, I can only get ~500K read IOPs and ~85K write IOPs.
On another much larger system with 72 cores and 4 NUMA nodes with 128G
of memory, I can only get ~310K read IOPs and ~175K write IOPs.
Look at the completion method (irqmode) and completion time
(completion_nsec).
OK, I found that queue_mode=0 (bio-based) is _much_ faster than blk-mq
(2, the default). Improving to ~950K read IOPs and ~675K write IOPs (on
the single numa node system).
Default for irqmode is 1 (softirq). 2 (timer) yields poor results. 0
(none) seems slightly slower than 1.
And if I use completion_nsec=1 I can bump up to ~990K read IOPs.
Seems the best, for IOPs, so far on this system is with:
modprobe null_blk gb=4 bs=4096 nr_devices=1 queue_mode=0 irqmode=1 completion_nsec=1 submit_queues=4
That sounds a bit odd. queue_mode=0 will always be a bit faster,
depending on how many threads, etc. But from 310K to 950K, that sounds
very suspicious.
And 500K/85K read/write is very low. Just did a quickie on a 2 node box
I have here, single thread performance with queue_mode=2 is around
500K/500K read/write.
--
Jens Axboe
--
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/dm-devel
