Hi,
QUEUE_FLAG_NOWAIT is set by default to mq drivers such as null_blk as
a part of QUEUE_FLAG_MQ_DEFAULT that gets assigned in following code
path see blk_mq_init_allocated_queue():-
null_add_dev()
if (dev->queue_mode == NULL_Q_MQ) {
blk_mq_alloc_disk()
__blk_mq_alloc_disk()
blk_mq_init_queue_data()
blk_mq_init_allocated_queue()
q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT;
}
But it not set when null_blk is loaded with NULL_Q_BIO mode in following
code path :-
if (dev->queue_mode == NULL_Q_BIO) {
nullb->disk = blk_alloc_disk(nullb->dev->home_node);
blk_alloc_disk()
blk_alloc_queue()
__alloc_disk_nodw()
}
Add a new module parameter nowait and respective configfs attr
that will set and clear the QUEUE_FLAG_NOWAIT based on a value set by
user in null_add_dev() irrespective of the queue mode, by default keep
it enabled to retain the original behaviour for the NULL_Q_MQ mode.
* Configfs Membacked mode:-
NULL_Q_BIO mode QUEUE_FLAG_NOWAIT disabled:-
---------------------------------------------
configfs-qmode-0-nowait-0-fio-1.log: read: IOPS=1295k, BW=5058MiB/s
configfs-qmode-0-nowait-0-fio-2.log: read: IOPS=1362k, BW=5318MiB/s
configfs-qmode-0-nowait-0-fio-3.log: read: IOPS=1332k, BW=5201MiB/s
NULL_Q_BIO mode QUEUE_FLAG_NOWAIT enabled :-
---------------------------------------------
configfs-qmode-0-nowait-1-fio-1.log: read: IOPS=2095k, BW=8183MiB/s
configfs-qmode-0-nowait-1-fio-2.log: read: IOPS=2085k, BW=8145MiB/s
configfs-qmode-0-nowait-1-fio-3.log: read: IOPS=2036k, BW=7955MiB/s
NULL_Q_MQ mode QUEUE_FLAG_NOWAIT disabled :-
---------------------------------------------
configfs-qmode-2-nowait-0-fio-1.log: read: IOPS=1288k, BW=5031MiB/s
configfs-qmode-2-nowait-0-fio-2.log: read: IOPS=1239k, BW=4839MiB/s
configfs-qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4889MiB/s
NULL_Q_MQ mode QUEUE_FLAG_NOWAIT enabled :-
---------------------------------------------
configfs-qmode-2-nowait-1-fio-1.log: read: IOPS=2101k, BW=8208MiB/s
configfs-qmode-2-nowait-1-fio-2.log: read: IOPS=2091k, BW=8169MiB/s
configfs-qmode-2-nowait-1-fio-3.log: read: IOPS=2088k, BW=8155MiB/s
* Non Configfs non-membacked mode:-
NULL_Q_BIO mode QUEUE_FLAG_NOWAIT disabled:-
---------------------------------------------
qmode-0-nowait-0-fio-1.log: read: IOPS=1362k, BW=5321MiB/s
qmode-0-nowait-0-fio-2.log: read: IOPS=1334k, BW=5210MiB/s
qmode-0-nowait-0-fio-3.log: read: IOPS=1386k, BW=5416MiB/s
NULL_Q_BIO mode QUEUE_FLAG_NOWAIT enabled :-
---------------------------------------------
qmode-0-nowait-1-fio-1.log: read: IOPS=5405k, BW=20.6GiB/s
qmode-0-nowait-1-fio-2.log: read: IOPS=5502k, BW=21.0GiB/s
qmode-0-nowait-1-fio-3.log: read: IOPS=5518k, BW=21.0GiB/s
NULL_Q_MQ mode QUEUE_FLAG_NOWAIT disabled :-
---------------------------------------------
qmode-2-nowait-0-fio-1.log: read: IOPS=1356k, BW=5296MiB/s
qmode-2-nowait-0-fio-2.log: read: IOPS=1318k, BW=5148MiB/s
qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4891MiB/s
NULL_Q_MQ mode QUEUE_FLAG_NOWAIT enabled :-
---------------------------------------------
qmode-2-nowait-1-fio-1.log: read: IOPS=5466k, BW=20.9GiB/s
qmode-2-nowait-1-fio-2.log: read: IOPS=5446k, BW=20.8GiB/s
qmode-2-nowait-1-fio-3.log: read: IOPS=5482k, BW=20.9GiB/s
Below is a performance test log for with and without configfs mambeacked
mode.
-ck
Chaitanya Kulkarni (1):
null_blk: allow user to set QUEUE_FLAG_NOWAIT
drivers/block/null_blk/main.c | 22 +++++++++++++++++-----
drivers/block/null_blk/null_blk.h | 1 +
2 files changed, 18 insertions(+), 5 deletions(-)
Performance test log for null_blk configfs and non-configfs:-
linux-block (for-next) # sh test-null-blk-no-wait.sh
+ modprobe -r null_blk
+ ./compile_nullb.sh
+ umount /mnt/nullb0
umount: /mnt/nullb0: not mounted.
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ dmesg -c
+ modprobe -r null_blk
+ lsmod
+ grep null_blk
++ nproc
+ make -j 48 M=drivers/block/ clean
CLEAN drivers/block/Module.symvers
++ nproc
+ make -j 48 M=drivers/block modules
CC [M] drivers/block/floppy.o
CC [M] drivers/block/brd.o
CC [M] drivers/block/loop.o
CC [M] drivers/block/nbd.o
CC [M] drivers/block/virtio_blk.o
CC [M] drivers/block/xen-blkfront.o
CC [M] drivers/block/rbd.o
CC [M] drivers/block/drbd/drbd_buildtag.o
CC [M] drivers/block/xen-blkback/blkback.o
CC [M] drivers/block/mtip32xx/mtip32xx.o
CC [M] drivers/block/zram/zcomp.o
CC [M] drivers/block/drbd/drbd_bitmap.o
CC [M] drivers/block/drbd/drbd_proc.o
CC [M] drivers/block/xen-blkback/xenbus.o
CC [M] drivers/block/zram/zram_drv.o
CC [M] drivers/block/drbd/drbd_worker.o
CC [M] drivers/block/null_blk/main.o
CC [M] drivers/block/drbd/drbd_receiver.o
CC [M] drivers/block/null_blk/trace.o
CC [M] drivers/block/null_blk/zoned.o
CC [M] drivers/block/drbd/drbd_req.o
CC [M] drivers/block/drbd/drbd_actlog.o
CC [M] drivers/block/drbd/drbd_main.o
CC [M] drivers/block/drbd/drbd_strings.o
CC [M] drivers/block/drbd/drbd_nl.o
CC [M] drivers/block/drbd/drbd_interval.o
CC [M] drivers/block/drbd/drbd_state.o
CC [M] drivers/block/drbd/drbd_nla.o
CC [M] drivers/block/drbd/drbd_debugfs.o
LD [M] drivers/block/zram/zram.o
LD [M] drivers/block/null_blk/null_blk.o
LD [M] drivers/block/xen-blkback/xen-blkback.o
LD [M] drivers/block/drbd/drbd.o
MODPOST drivers/block/Module.symvers
CC [M] drivers/block/floppy.mod.o
CC [M] drivers/block/brd.mod.o
CC [M] drivers/block/loop.mod.o
CC [M] drivers/block/nbd.mod.o
CC [M] drivers/block/virtio_blk.mod.o
CC [M] drivers/block/xen-blkfront.mod.o
CC [M] drivers/block/xen-blkback/xen-blkback.mod.o
CC [M] drivers/block/drbd/drbd.mod.o
CC [M] drivers/block/rbd.mod.o
CC [M] drivers/block/mtip32xx/mtip32xx.mod.o
CC [M] drivers/block/zram/zram.mod.o
CC [M] drivers/block/null_blk/null_blk.mod.o
LD [M] drivers/block/xen-blkfront.ko
LD [M] drivers/block/brd.ko
LD [M] drivers/block/xen-blkback/xen-blkback.ko
LD [M] drivers/block/loop.ko
LD [M] drivers/block/virtio_blk.ko
LD [M] drivers/block/mtip32xx/mtip32xx.ko
LD [M] drivers/block/rbd.ko
LD [M] drivers/block/zram/zram.ko
LD [M] drivers/block/nbd.ko
LD [M] drivers/block/drbd/drbd.ko
LD [M] drivers/block/floppy.ko
LD [M] drivers/block/null_blk/null_blk.ko
+ HOST=drivers/block/null_blk/
++ uname -r
+ HOST_DEST=/lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk/
+ cp drivers/block/null_blk//null_blk.ko /lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk//
+ ls -lrth /lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk//null_blk.ko
-rw-r--r--. 1 root root 1.2M Apr 12 01:01 /lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk//null_blk.ko
+ sleep 1
+ dmesg -c
+ test_non_configfs
+ modprobe null_blk nowait=0 queue_mode=0
+ dmesg -c
[ 9509.969343] null_blk: disk nullb0 created
[ 9509.969347] null_blk: module loaded
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-0-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=5415MiB/s][r=1386k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-0-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=5184MiB/s][r=1327k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-0-fio-3.log
+ modprobe -r null_blk)][100.0%][r=5525MiB/s][r=1414k IOPS][eta 00m:00s]
+ modprobe null_blk nowait=1 queue_mode=0
+ dmesg -c
[ 9570.634724] null_blk: disk nullb0 created
[ 9570.634729] null_blk: module loaded
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-1-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=20.8GiB/s][r=5452k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-1-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=21.3GiB/s][r=5577k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-1-fio-3.log
+ modprobe -r null_blk)][100.0%][r=21.1GiB/s][r=5536k IOPS][eta 00m:00s]
+ modprobe null_blk nowait=0 queue_mode=2
+ dmesg -c
[ 9631.320182] null_blk: disk nullb0 created
[ 9631.320186] null_blk: module loaded
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-0-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=6132MiB/s][r=1570k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-0-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=5554MiB/s][r=1422k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-0-fio-3.log
+ modprobe -r null_blk)][100.0%][r=5218MiB/s][r=1336k IOPS][eta 00m:00s]
+ modprobe null_blk nowait=1 queue_mode=2
+ dmesg -c
[ 9691.986330] null_blk: disk nullb0 created
[ 9691.986333] null_blk: module loaded
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-1-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=21.2GiB/s][r=5547k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-1-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=20.3GiB/s][r=5313k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-1-fio-3.log
+ modprobe -r null_blk)][100.0%][r=21.2GiB/s][r=5547k IOPS][eta 00m:00s]
+ test_configfs 0 0
+ qmode=0
+ nowait=0
+ modprobe null_blk nr_devices=0
+ NULLB_DIR=config/nullb/nullb0
+ mkdir config/nullb/nullb0
+ echo 1
+ echo 4096
+ echo 20480
+ echo 0
+ echo 0
+ echo 1
+ '[' 0 -ne 0 ']'
+ sleep .50
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-0-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=5174MiB/s][r=1325k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-0-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=5904MiB/s][r=1512k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-0-fio-3.log
+ rmdir config/nullb/nullb00.0%][r=5493MiB/s][r=1406k IOPS][eta 00m:00s]
+ modprobe -r null_blk
+ test_configfs 0 1
+ qmode=0
+ nowait=1
+ modprobe null_blk nr_devices=0
+ NULLB_DIR=config/nullb/nullb0
+ mkdir config/nullb/nullb0
+ echo 1
+ echo 4096
+ echo 20480
+ echo 0
+ echo 1
+ echo 1
+ '[' 0 -ne 0 ']'
+ sleep .50
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-1-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=8265MiB/s][r=2116k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-1-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=8090MiB/s][r=2071k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-1-fio-3.log
+ rmdir config/nullb/nullb00.0%][r=7971MiB/s][r=2041k IOPS][eta 00m:00s]
+ modprobe -r null_blk
+ test_configfs 2 0
+ qmode=2
+ nowait=0
+ modprobe null_blk nr_devices=0
+ NULLB_DIR=config/nullb/nullb0
+ mkdir config/nullb/nullb0
+ echo 1
+ echo 4096
+ echo 20480
+ echo 2
+ echo 0
+ echo 1
+ '[' 0 -ne 0 ']'
+ sleep .50
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-0-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=5247MiB/s][r=1343k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-0-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=5132MiB/s][r=1314k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-0-fio-3.log
+ rmdir config/nullb/nullb00.0%][r=4960MiB/s][r=1270k IOPS][eta 00m:00s]
+ modprobe -r null_blk
+ test_configfs 2 1
+ qmode=2
+ nowait=1
+ modprobe null_blk nr_devices=0
+ NULLB_DIR=config/nullb/nullb0
+ mkdir config/nullb/nullb0
+ echo 1
+ echo 4096
+ echo 20480
+ echo 2
+ echo 1
+ echo 1
+ '[' 0 -ne 0 ']'
+ sleep .50
+ for i in 1 2 3
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-1-fio-1.log
+ for i in 1 2 3 [r(24)][100.0%][r=8220MiB/s][r=2104k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-1-fio-2.log
+ for i in 1 2 3 [r(24)][100.0%][r=8257MiB/s][r=2114k IOPS][eta 00m:00s]
+ fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-1-fio-3.log
+ rmdir config/nullb/nullb00.0%][r=8163MiB/s][r=2090k IOPS][eta 00m:00s]
+ modprobe -r null_blk
linux-block (for-next) #
linux-block (for-next) #
linux-block (for-next) #
linux-block (for-next) #
linux-block (for-next) # grep IOPS *fio*log
configfs-qmode-0-nowait-0-fio-1.log: read: IOPS=1295k, BW=5058MiB/s (5303MB/s)(98.8GiB/20002msec)
configfs-qmode-0-nowait-0-fio-2.log: read: IOPS=1362k, BW=5318MiB/s (5577MB/s)(104GiB/20002msec)
configfs-qmode-0-nowait-0-fio-3.log: read: IOPS=1332k, BW=5201MiB/s (5454MB/s)(102GiB/20001msec)
configfs-qmode-0-nowait-1-fio-1.log: read: IOPS=2095k, BW=8183MiB/s (8581MB/s)(160GiB/20001msec)
configfs-qmode-0-nowait-1-fio-2.log: read: IOPS=2085k, BW=8145MiB/s (8541MB/s)(159GiB/20002msec)
configfs-qmode-0-nowait-1-fio-3.log: read: IOPS=2036k, BW=7955MiB/s (8341MB/s)(155GiB/20001msec)
configfs-qmode-2-nowait-0-fio-1.log: read: IOPS=1288k, BW=5031MiB/s (5275MB/s)(98.3GiB/20002msec)
configfs-qmode-2-nowait-0-fio-2.log: read: IOPS=1239k, BW=4839MiB/s (5074MB/s)(94.5GiB/20001msec)
configfs-qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4889MiB/s (5127MB/s)(95.5GiB/20001msec)
configfs-qmode-2-nowait-1-fio-1.log: read: IOPS=2101k, BW=8208MiB/s (8607MB/s)(160GiB/20001msec)
configfs-qmode-2-nowait-1-fio-2.log: read: IOPS=2091k, BW=8169MiB/s (8565MB/s)(160GiB/20001msec)
configfs-qmode-2-nowait-1-fio-3.log: read: IOPS=2088k, BW=8155MiB/s (8551MB/s)(159GiB/20001msec)
qmode-0-nowait-0-fio-1.log: read: IOPS=1362k, BW=5321MiB/s (5579MB/s)(104GiB/20001msec)
qmode-0-nowait-0-fio-2.log: read: IOPS=1334k, BW=5210MiB/s (5463MB/s)(102GiB/20001msec)
qmode-0-nowait-0-fio-3.log: read: IOPS=1386k, BW=5416MiB/s (5679MB/s)(106GiB/20001msec)
qmode-0-nowait-1-fio-1.log: read: IOPS=5405k, BW=20.6GiB/s (22.1GB/s)(412GiB/20002msec)
qmode-0-nowait-1-fio-2.log: read: IOPS=5502k, BW=21.0GiB/s (22.5GB/s)(420GiB/20001msec)
qmode-0-nowait-1-fio-3.log: read: IOPS=5518k, BW=21.0GiB/s (22.6GB/s)(421GiB/20002msec)
qmode-2-nowait-0-fio-1.log: read: IOPS=1356k, BW=5296MiB/s (5554MB/s)(103GiB/20001msec)
qmode-2-nowait-0-fio-2.log: read: IOPS=1318k, BW=5148MiB/s (5398MB/s)(101GiB/20001msec)
qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4891MiB/s (5128MB/s)(95.5GiB/20001msec)
qmode-2-nowait-1-fio-1.log: read: IOPS=5466k, BW=20.9GiB/s (22.4GB/s)(417GiB/20001msec)
qmode-2-nowait-1-fio-2.log: read: IOPS=5446k, BW=20.8GiB/s (22.3GB/s)(416GiB/20002msec)
qmode-2-nowait-1-fio-3.log: read: IOPS=5482k, BW=20.9GiB/s (22.5GB/s)(418GiB/20002msec)
--
2.29.0
