+ dlemoal@xxxxxxxxxx.
On 4/12/23 01:47, Chaitanya Kulkarni wrote:
> Hi,
>
> QUEUE_FLAG_NOWAIT is set by default to mq drivers such as null_blk as
> a part of QUEUE_FLAG_MQ_DEFAULT that gets assigned in following code
> path see blk_mq_init_allocated_queue():-
>
> null_add_dev()
> if (dev->queue_mode == NULL_Q_MQ) {
> blk_mq_alloc_disk()
> __blk_mq_alloc_disk()
> blk_mq_init_queue_data()
> blk_mq_init_allocated_queue()
> q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT;
> }
>
> But it not set when null_blk is loaded with NULL_Q_BIO mode in following
> code path :-
>
> if (dev->queue_mode == NULL_Q_BIO) {
> nullb->disk = blk_alloc_disk(nullb->dev->home_node);
> blk_alloc_disk()
> blk_alloc_queue()
> __alloc_disk_nodw()
> }
>
> Add a new module parameter nowait and respective configfs attr
> that will set and clear the QUEUE_FLAG_NOWAIT based on a value set by
> user in null_add_dev() irrespective of the queue mode, by default keep
> it enabled to retain the original behaviour for the NULL_Q_MQ mode.
>
> * Configfs Membacked mode:-
>
> NULL_Q_BIO mode QUEUE_FLAG_NOWAIT disabled:-
> ---------------------------------------------
> configfs-qmode-0-nowait-0-fio-1.log: read: IOPS=1295k, BW=5058MiB/s
> configfs-qmode-0-nowait-0-fio-2.log: read: IOPS=1362k, BW=5318MiB/s
> configfs-qmode-0-nowait-0-fio-3.log: read: IOPS=1332k, BW=5201MiB/s
>
> NULL_Q_BIO mode QUEUE_FLAG_NOWAIT enabled :-
> ---------------------------------------------
> configfs-qmode-0-nowait-1-fio-1.log: read: IOPS=2095k, BW=8183MiB/s
> configfs-qmode-0-nowait-1-fio-2.log: read: IOPS=2085k, BW=8145MiB/s
> configfs-qmode-0-nowait-1-fio-3.log: read: IOPS=2036k, BW=7955MiB/s
>
> NULL_Q_MQ mode QUEUE_FLAG_NOWAIT disabled :-
> ---------------------------------------------
> configfs-qmode-2-nowait-0-fio-1.log: read: IOPS=1288k, BW=5031MiB/s
> configfs-qmode-2-nowait-0-fio-2.log: read: IOPS=1239k, BW=4839MiB/s
> configfs-qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4889MiB/s
>
> NULL_Q_MQ mode QUEUE_FLAG_NOWAIT enabled :-
> ---------------------------------------------
> configfs-qmode-2-nowait-1-fio-1.log: read: IOPS=2101k, BW=8208MiB/s
> configfs-qmode-2-nowait-1-fio-2.log: read: IOPS=2091k, BW=8169MiB/s
> configfs-qmode-2-nowait-1-fio-3.log: read: IOPS=2088k, BW=8155MiB/s
>
> * Non Configfs non-membacked mode:-
>
> NULL_Q_BIO mode QUEUE_FLAG_NOWAIT disabled:-
> ---------------------------------------------
> qmode-0-nowait-0-fio-1.log: read: IOPS=1362k, BW=5321MiB/s
> qmode-0-nowait-0-fio-2.log: read: IOPS=1334k, BW=5210MiB/s
> qmode-0-nowait-0-fio-3.log: read: IOPS=1386k, BW=5416MiB/s
>
> NULL_Q_BIO mode QUEUE_FLAG_NOWAIT enabled :-
> ---------------------------------------------
> qmode-0-nowait-1-fio-1.log: read: IOPS=5405k, BW=20.6GiB/s
> qmode-0-nowait-1-fio-2.log: read: IOPS=5502k, BW=21.0GiB/s
> qmode-0-nowait-1-fio-3.log: read: IOPS=5518k, BW=21.0GiB/s
>
> NULL_Q_MQ mode QUEUE_FLAG_NOWAIT disabled :-
> ---------------------------------------------
> qmode-2-nowait-0-fio-1.log: read: IOPS=1356k, BW=5296MiB/s
> qmode-2-nowait-0-fio-2.log: read: IOPS=1318k, BW=5148MiB/s
> qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4891MiB/s
>
> NULL_Q_MQ mode QUEUE_FLAG_NOWAIT enabled :-
> ---------------------------------------------
> qmode-2-nowait-1-fio-1.log: read: IOPS=5466k, BW=20.9GiB/s
> qmode-2-nowait-1-fio-2.log: read: IOPS=5446k, BW=20.8GiB/s
> qmode-2-nowait-1-fio-3.log: read: IOPS=5482k, BW=20.9GiB/s
>
> Below is a performance test log for with and without configfs mambeacked
> mode.
>
> -ck
>
> Chaitanya Kulkarni (1):
> null_blk: allow user to set QUEUE_FLAG_NOWAIT
>
> drivers/block/null_blk/main.c | 22 +++++++++++++++++-----
> drivers/block/null_blk/null_blk.h | 1 +
> 2 files changed, 18 insertions(+), 5 deletions(-)
>
> Performance test log for null_blk configfs and non-configfs:-
>
> linux-block (for-next) # sh test-null-blk-no-wait.sh
> + modprobe -r null_blk
> + ./compile_nullb.sh
> + umount /mnt/nullb0
> umount: /mnt/nullb0: not mounted.
> + rmdir 'config/nullb/nullb*'
> rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
> + dmesg -c
> + modprobe -r null_blk
> + lsmod
> + grep null_blk
> ++ nproc
> + make -j 48 M=drivers/block/ clean
> CLEAN drivers/block/Module.symvers
> ++ nproc
> + make -j 48 M=drivers/block modules
> CC [M] drivers/block/floppy.o
> CC [M] drivers/block/brd.o
> CC [M] drivers/block/loop.o
> CC [M] drivers/block/nbd.o
> CC [M] drivers/block/virtio_blk.o
> CC [M] drivers/block/xen-blkfront.o
> CC [M] drivers/block/rbd.o
> CC [M] drivers/block/drbd/drbd_buildtag.o
> CC [M] drivers/block/xen-blkback/blkback.o
> CC [M] drivers/block/mtip32xx/mtip32xx.o
> CC [M] drivers/block/zram/zcomp.o
> CC [M] drivers/block/drbd/drbd_bitmap.o
> CC [M] drivers/block/drbd/drbd_proc.o
> CC [M] drivers/block/xen-blkback/xenbus.o
> CC [M] drivers/block/zram/zram_drv.o
> CC [M] drivers/block/drbd/drbd_worker.o
> CC [M] drivers/block/null_blk/main.o
> CC [M] drivers/block/drbd/drbd_receiver.o
> CC [M] drivers/block/null_blk/trace.o
> CC [M] drivers/block/null_blk/zoned.o
> CC [M] drivers/block/drbd/drbd_req.o
> CC [M] drivers/block/drbd/drbd_actlog.o
> CC [M] drivers/block/drbd/drbd_main.o
> CC [M] drivers/block/drbd/drbd_strings.o
> CC [M] drivers/block/drbd/drbd_nl.o
> CC [M] drivers/block/drbd/drbd_interval.o
> CC [M] drivers/block/drbd/drbd_state.o
> CC [M] drivers/block/drbd/drbd_nla.o
> CC [M] drivers/block/drbd/drbd_debugfs.o
> LD [M] drivers/block/zram/zram.o
> LD [M] drivers/block/null_blk/null_blk.o
> LD [M] drivers/block/xen-blkback/xen-blkback.o
> LD [M] drivers/block/drbd/drbd.o
> MODPOST drivers/block/Module.symvers
> CC [M] drivers/block/floppy.mod.o
> CC [M] drivers/block/brd.mod.o
> CC [M] drivers/block/loop.mod.o
> CC [M] drivers/block/nbd.mod.o
> CC [M] drivers/block/virtio_blk.mod.o
> CC [M] drivers/block/xen-blkfront.mod.o
> CC [M] drivers/block/xen-blkback/xen-blkback.mod.o
> CC [M] drivers/block/drbd/drbd.mod.o
> CC [M] drivers/block/rbd.mod.o
> CC [M] drivers/block/mtip32xx/mtip32xx.mod.o
> CC [M] drivers/block/zram/zram.mod.o
> CC [M] drivers/block/null_blk/null_blk.mod.o
> LD [M] drivers/block/xen-blkfront.ko
> LD [M] drivers/block/brd.ko
> LD [M] drivers/block/xen-blkback/xen-blkback.ko
> LD [M] drivers/block/loop.ko
> LD [M] drivers/block/virtio_blk.ko
> LD [M] drivers/block/mtip32xx/mtip32xx.ko
> LD [M] drivers/block/rbd.ko
> LD [M] drivers/block/zram/zram.ko
> LD [M] drivers/block/nbd.ko
> LD [M] drivers/block/drbd/drbd.ko
> LD [M] drivers/block/floppy.ko
> LD [M] drivers/block/null_blk/null_blk.ko
> + HOST=drivers/block/null_blk/
> ++ uname -r
> + HOST_DEST=/lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk/
> + cp drivers/block/null_blk//null_blk.ko /lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk//
> + ls -lrth /lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk//null_blk.ko
> -rw-r--r--. 1 root root 1.2M Apr 12 01:01 /lib/modules/6.3.0-rc5lblk+/kernel/drivers/block/null_blk//null_blk.ko
> + sleep 1
> + dmesg -c
> + test_non_configfs
> + modprobe null_blk nowait=0 queue_mode=0
> + dmesg -c
> [ 9509.969343] null_blk: disk nullb0 created
> [ 9509.969347] null_blk: module loaded
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-0-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=5415MiB/s][r=1386k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-0-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=5184MiB/s][r=1327k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-0-fio-3.log
> + modprobe -r null_blk)][100.0%][r=5525MiB/s][r=1414k IOPS][eta 00m:00s]
> + modprobe null_blk nowait=1 queue_mode=0
> + dmesg -c
> [ 9570.634724] null_blk: disk nullb0 created
> [ 9570.634729] null_blk: module loaded
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-1-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=20.8GiB/s][r=5452k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-1-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=21.3GiB/s][r=5577k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-0-nowait-1-fio-3.log
> + modprobe -r null_blk)][100.0%][r=21.1GiB/s][r=5536k IOPS][eta 00m:00s]
> + modprobe null_blk nowait=0 queue_mode=2
> + dmesg -c
> [ 9631.320182] null_blk: disk nullb0 created
> [ 9631.320186] null_blk: module loaded
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-0-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=6132MiB/s][r=1570k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-0-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=5554MiB/s][r=1422k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-0-fio-3.log
> + modprobe -r null_blk)][100.0%][r=5218MiB/s][r=1336k IOPS][eta 00m:00s]
> + modprobe null_blk nowait=1 queue_mode=2
> + dmesg -c
> [ 9691.986330] null_blk: disk nullb0 created
> [ 9691.986333] null_blk: module loaded
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-1-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=21.2GiB/s][r=5547k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-1-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=20.3GiB/s][r=5313k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=qmode-2-nowait-1-fio-3.log
> + modprobe -r null_blk)][100.0%][r=21.2GiB/s][r=5547k IOPS][eta 00m:00s]
> + test_configfs 0 0
> + qmode=0
> + nowait=0
> + modprobe null_blk nr_devices=0
> + NULLB_DIR=config/nullb/nullb0
> + mkdir config/nullb/nullb0
> + echo 1
> + echo 4096
> + echo 20480
> + echo 0
> + echo 0
> + echo 1
> + '[' 0 -ne 0 ']'
> + sleep .50
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-0-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=5174MiB/s][r=1325k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-0-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=5904MiB/s][r=1512k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-0-fio-3.log
> + rmdir config/nullb/nullb00.0%][r=5493MiB/s][r=1406k IOPS][eta 00m:00s]
> + modprobe -r null_blk
> + test_configfs 0 1
> + qmode=0
> + nowait=1
> + modprobe null_blk nr_devices=0
> + NULLB_DIR=config/nullb/nullb0
> + mkdir config/nullb/nullb0
> + echo 1
> + echo 4096
> + echo 20480
> + echo 0
> + echo 1
> + echo 1
> + '[' 0 -ne 0 ']'
> + sleep .50
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-1-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=8265MiB/s][r=2116k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-1-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=8090MiB/s][r=2071k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-0-nowait-1-fio-3.log
> + rmdir config/nullb/nullb00.0%][r=7971MiB/s][r=2041k IOPS][eta 00m:00s]
> + modprobe -r null_blk
> + test_configfs 2 0
> + qmode=2
> + nowait=0
> + modprobe null_blk nr_devices=0
> + NULLB_DIR=config/nullb/nullb0
> + mkdir config/nullb/nullb0
> + echo 1
> + echo 4096
> + echo 20480
> + echo 2
> + echo 0
> + echo 1
> + '[' 0 -ne 0 ']'
> + sleep .50
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-0-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=5247MiB/s][r=1343k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-0-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=5132MiB/s][r=1314k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-0-fio-3.log
> + rmdir config/nullb/nullb00.0%][r=4960MiB/s][r=1270k IOPS][eta 00m:00s]
> + modprobe -r null_blk
> + test_configfs 2 1
> + qmode=2
> + nowait=1
> + modprobe null_blk nr_devices=0
> + NULLB_DIR=config/nullb/nullb0
> + mkdir config/nullb/nullb0
> + echo 1
> + echo 4096
> + echo 20480
> + echo 2
> + echo 1
> + echo 1
> + '[' 0 -ne 0 ']'
> + sleep .50
> + for i in 1 2 3
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-1-fio-1.log
> + for i in 1 2 3 [r(24)][100.0%][r=8220MiB/s][r=2104k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-1-fio-2.log
> + for i in 1 2 3 [r(24)][100.0%][r=8257MiB/s][r=2114k IOPS][eta 00m:00s]
> + fio fio/randread.fio --filename=/dev/nullb0 --output=configfs-qmode-2-nowait-1-fio-3.log
> + rmdir config/nullb/nullb00.0%][r=8163MiB/s][r=2090k IOPS][eta 00m:00s]
> + modprobe -r null_blk
> linux-block (for-next) #
> linux-block (for-next) #
> linux-block (for-next) #
> linux-block (for-next) #
> linux-block (for-next) # grep IOPS *fio*log
> configfs-qmode-0-nowait-0-fio-1.log: read: IOPS=1295k, BW=5058MiB/s (5303MB/s)(98.8GiB/20002msec)
> configfs-qmode-0-nowait-0-fio-2.log: read: IOPS=1362k, BW=5318MiB/s (5577MB/s)(104GiB/20002msec)
> configfs-qmode-0-nowait-0-fio-3.log: read: IOPS=1332k, BW=5201MiB/s (5454MB/s)(102GiB/20001msec)
> configfs-qmode-0-nowait-1-fio-1.log: read: IOPS=2095k, BW=8183MiB/s (8581MB/s)(160GiB/20001msec)
> configfs-qmode-0-nowait-1-fio-2.log: read: IOPS=2085k, BW=8145MiB/s (8541MB/s)(159GiB/20002msec)
> configfs-qmode-0-nowait-1-fio-3.log: read: IOPS=2036k, BW=7955MiB/s (8341MB/s)(155GiB/20001msec)
> configfs-qmode-2-nowait-0-fio-1.log: read: IOPS=1288k, BW=5031MiB/s (5275MB/s)(98.3GiB/20002msec)
> configfs-qmode-2-nowait-0-fio-2.log: read: IOPS=1239k, BW=4839MiB/s (5074MB/s)(94.5GiB/20001msec)
> configfs-qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4889MiB/s (5127MB/s)(95.5GiB/20001msec)
> configfs-qmode-2-nowait-1-fio-1.log: read: IOPS=2101k, BW=8208MiB/s (8607MB/s)(160GiB/20001msec)
> configfs-qmode-2-nowait-1-fio-2.log: read: IOPS=2091k, BW=8169MiB/s (8565MB/s)(160GiB/20001msec)
> configfs-qmode-2-nowait-1-fio-3.log: read: IOPS=2088k, BW=8155MiB/s (8551MB/s)(159GiB/20001msec)
> qmode-0-nowait-0-fio-1.log: read: IOPS=1362k, BW=5321MiB/s (5579MB/s)(104GiB/20001msec)
> qmode-0-nowait-0-fio-2.log: read: IOPS=1334k, BW=5210MiB/s (5463MB/s)(102GiB/20001msec)
> qmode-0-nowait-0-fio-3.log: read: IOPS=1386k, BW=5416MiB/s (5679MB/s)(106GiB/20001msec)
> qmode-0-nowait-1-fio-1.log: read: IOPS=5405k, BW=20.6GiB/s (22.1GB/s)(412GiB/20002msec)
> qmode-0-nowait-1-fio-2.log: read: IOPS=5502k, BW=21.0GiB/s (22.5GB/s)(420GiB/20001msec)
> qmode-0-nowait-1-fio-3.log: read: IOPS=5518k, BW=21.0GiB/s (22.6GB/s)(421GiB/20002msec)
> qmode-2-nowait-0-fio-1.log: read: IOPS=1356k, BW=5296MiB/s (5554MB/s)(103GiB/20001msec)
> qmode-2-nowait-0-fio-2.log: read: IOPS=1318k, BW=5148MiB/s (5398MB/s)(101GiB/20001msec)
> qmode-2-nowait-0-fio-3.log: read: IOPS=1252k, BW=4891MiB/s (5128MB/s)(95.5GiB/20001msec)
> qmode-2-nowait-1-fio-1.log: read: IOPS=5466k, BW=20.9GiB/s (22.4GB/s)(417GiB/20001msec)
> qmode-2-nowait-1-fio-2.log: read: IOPS=5446k, BW=20.8GiB/s (22.3GB/s)(416GiB/20002msec)
> qmode-2-nowait-1-fio-3.log: read: IOPS=5482k, BW=20.9GiB/s (22.5GB/s)(418GiB/20002msec)
>
