Hi, This adds support for the REQ_OP_VERIFY. In this version we add support for block layer. NVMe host side, NVMeOF Block device backend, and NVMeOF File device backend and null_blk driver. In this version we also add a new blkverify userspace tool along with the testcases patch for the util-linux, this patch will be followed by the this series. Below is the summary of testlog :- 1. NVMeOF bdev-ns null_blk verify=0 (triggering bdev emulation code) :- ----------------------------------------------------------------------- linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1 linux-block (for-next) # dmesg -c [ 1171.171536] nvmet: nvmet_bdev_emulate_verify_work 467 linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1 NVME Verify Success linux-block (for-next) # dmesg -c [ 1199.322161] nvmet: nvmet_bdev_emulate_verify_work 467 2. NVMeOF bdev-ns null_blk verify=1. ----------------------------------------------------------------------- linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1 linux-block (for-next) # dmesg -c [ 1257.661548] nvmet: nvmet_bdev_execute_verify 506 [ 1257.661558] null_blk: null_process_cmd 1406 linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1 NVME Verify Success linux-block (for-next) # dmesg -c [ 1269.613415] nvmet: nvmet_bdev_execute_verify 506 [ 1269.613425] null_blk: null_process_cmd 1406 3. NVMeOF file-ns :- ----------------------------------------------------------------------- linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1 linux-block (for-next) # dmesg -c [ 3452.675959] nvme_setup_verify 844 [ 3452.675969] nvmet: nvmet_file_execute_verify 525 [ 3452.675971] nvmet: nvmet_file_emulate_verify_work 502 [ 3452.675972] nvmet: do_direct_io_emulate_verify 431 linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1 NVME Verify Success linux-block (for-next) # dmesg -c [ 3459.794385] nvmet: nvmet_file_execute_verify 525 [ 3459.794389] nvmet: nvmet_file_emulate_verify_work 502 [ 3459.794391] nvmet: do_direct_io_emulate_verify 431 4. NVMe PCIe device. ----------------------------------------------------------------------- linux-block (for-next) # modprobe nvme linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme0n1 linux-block (for-next) # dmesg -c [ 2763.432194] nvme nvme0: pci function 0000:00:04.0 [ 2763.473827] nvme nvme0: 48/0/0 default/read/poll queues [ 2763.478868] nvme nvme0: Ignoring bogus Namespace Identifiers [ 2766.583923] nvme_setup_verify 844 Here is a link for the complete cover-letter for the background to save reviewer's time :- https://patchwork.kernel.org/project/dm-devel/cover/20211104064634.4481-1-chaitanyak@xxxxxxxxxx/ -ck Chaitanya Kulkarni (6): block: add support for REQ_OP_VERIFY nvme: add support for the Verify command nvmet: add Verify command support for bdev-ns nvmet: add Verify emulation support for bdev-ns nvmet: add verify emulation support for file-ns null_blk: add REQ_OP_VERIFY support Documentation/ABI/stable/sysfs-block | 12 +++ block/blk-core.c | 5 + block/blk-lib.c | 155 +++++++++++++++++++++++++++ block/blk-merge.c | 18 ++++ block/blk-settings.c | 17 +++ block/blk-sysfs.c | 8 ++ block/blk.h | 4 + block/ioctl.c | 35 ++++++ drivers/block/null_blk/main.c | 20 +++- drivers/block/null_blk/null_blk.h | 1 + drivers/nvme/host/core.c | 33 ++++++ drivers/nvme/target/admin-cmd.c | 3 +- drivers/nvme/target/core.c | 14 ++- drivers/nvme/target/io-cmd-bdev.c | 75 +++++++++++++ drivers/nvme/target/io-cmd-file.c | 152 ++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 4 +- include/linux/bio.h | 9 +- include/linux/blk_types.h | 2 + include/linux/blkdev.h | 22 ++++ include/linux/nvme.h | 19 ++++ include/uapi/linux/fs.h | 1 + 21 files changed, 601 insertions(+), 8 deletions(-) -- 2.29.0 ######################## NVMeOF bdev-ns null_blk verify=0 ###################### 0 directories, 0 files linux-block (for-next) # ./bdev.sh 1 ++ FILE=/dev/nvme0n1 ++ NN=1 ++ NQN=testnqn ++ let NR_DEVICES=NN+1 ++ modprobe -r null_blk ++ modprobe null_blk nr_devices=0 verify=0 ++ modprobe nvme ++ modprobe nvme-fabrics ++ modprobe nvmet ++ modprobe nvme-loop ++ dmesg -c ++ sleep 2 ++ tree /sys/kernel/config /sys/kernel/config ├── nullb │ └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn ++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop ++ echo -n 1 ++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ sleep 1 ++ echo transport=loop,nqn=testnqn +++ shuf -i 1-1 -n 1 ++ for i in `shuf -i 1-$NN -n $NN` ++ mkdir config/nullb/nullb1 ++ echo 4096 ++ echo 20971520 ++ echo 1 +++ cat config/nullb/nullb1/index ++ IDX=0 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ let IDX=IDX+1 ++ echo ' ####### /dev/nullb1' ####### /dev/nullb1 ++ echo -n /dev/nullb1 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb1 ++ echo 1 ++ dmesg -c [ 1150.985720] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:7f5b83f1-b258-4300-9a55-cd1902bea8c2. [ 1150.985882] nvme nvme1: creating 48 I/O queues. [ 1150.990654] nvme nvme1: new ctrl: "testnqn" [ 1150.995681] null_blk: disk nullb1 created [ 1150.998886] nvmet: adding nsid 1 to subsystem testnqn [ 1151.000716] nvme nvme1: rescanning namespaces. ++ sleep 1 ++ mount ++ column -t ++ grep nvme ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 /dev/nvme1n1 ++ cnt=2 ++ echo 2 2 ++ '[' 2 -gt 1 ']' ++ break ++ dmesg -c linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1 linux-block (for-next) # dmesg -c [ 1171.171536] nvmet: nvmet_bdev_emulate_verify_work 467 linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1 NVME Verify Success linux-block (for-next) # dmesg -c [ 1199.322161] nvmet: nvmet_bdev_emulate_verify_work 467 linux-block (for-next) # ./delete.sh 1 + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.343s user 0m0.002s sys 0m0.003s ++ shuf -i 1-1 -n 1 + for i in `shuf -i 1-$NN -n $NN` + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rmdir config/nullb/nullb1 + sleep 2 + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + sleep 1 + rmdir /sys/kernel/config/nvmet/ports/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + sleep 1 + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files ######################## NVMeOF bdev-ns null_blk verify=1 ##################### linux-block (for-next) # ./bdev.sh 1 ++ FILE=/dev/nvme0n1 ++ NN=1 ++ NQN=testnqn ++ let NR_DEVICES=NN+1 ++ modprobe -r null_blk ++ modprobe null_blk nr_devices=0 verify=1 ++ modprobe nvme ++ modprobe nvme-fabrics ++ modprobe nvmet ++ modprobe nvme-loop ++ dmesg -c ++ sleep 2 ++ tree /sys/kernel/config /sys/kernel/config ├── nullb │ └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn ++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop ++ echo -n 1 ++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ sleep 1 ++ echo transport=loop,nqn=testnqn +++ shuf -i 1-1 -n 1 ++ for i in `shuf -i 1-$NN -n $NN` ++ mkdir config/nullb/nullb1 ++ echo 4096 ++ echo 20971520 ++ echo 1 +++ cat config/nullb/nullb1/index ++ IDX=0 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ let IDX=IDX+1 ++ echo ' ####### /dev/nullb1' ####### /dev/nullb1 ++ echo -n /dev/nullb1 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb1 ++ echo 1 ++ dmesg -c [ 1250.311632] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:328bb18d-3662-48eb-8bc2-ca7d4ad73799. [ 1250.311853] nvme nvme1: creating 48 I/O queues. [ 1250.316251] nvme nvme1: new ctrl: "testnqn" [ 1250.321710] null_blk: disk nullb1 created [ 1250.324678] nvmet: adding nsid 1 to subsystem testnqn [ 1250.326546] nvme nvme1: rescanning namespaces. ++ sleep 1 ++ mount ++ column -t ++ grep nvme ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 /dev/nvme1n1 ++ cnt=2 ++ echo 2 2 ++ '[' 2 -gt 1 ']' ++ break ++ dmesg -c linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1 linux-block (for-next) # dmesg -c [ 1257.661548] nvmet: nvmet_bdev_execute_verify 506 [ 1257.661558] null_blk: null_process_cmd 1406 linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1 NVME Verify Success linux-block (for-next) # dmesg -c [ 1269.613415] nvmet: nvmet_bdev_execute_verify 506 [ 1269.613425] null_blk: null_process_cmd 1406 linux-block (for-next) # linux-block (for-next) # ./delete.sh 1 + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.339s user 0m0.002s sys 0m0.003s ++ shuf -i 1-1 -n 1 + for i in `shuf -i 1-$NN -n $NN` + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rmdir config/nullb/nullb1 + sleep 2 + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + sleep 1 + rmdir /sys/kernel/config/nvmet/ports/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + sleep 1 + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files ################################### NVMeOF file-ns ############################# linux-block (for-next) # linux-block (for-next) # ./file.sh 1 ++ FILE=/mnt/backend/nvme1n1 ++ SS=testnqn ++ SSPATH=/sys/kernel/config/nvmet/subsystems/testnqn/ ++ PORTS=/sys/kernel/config/nvmet/ports ++ main ++ load_modules ++ dmesg -c ++ modprobe nvme ++ modprobe nvme-fabrics ++ modprobe nvmet ++ modprobe nvme-loop ++ sleep 3 ++ mount_fs ++ make_nullb ++ ./compile_nullb.sh + umount /mnt/nullb0 umount: /mnt/nullb0: not mounted. + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + dmesg -c + modprobe -r null_blk + lsmod + grep null_blk ++ nproc + make -j 48 M=drivers/block modules + HOST=drivers/block/null_blk/ ++ uname -r + HOST_DEST=/lib/modules/5.18.0blk+/kernel/drivers/block/null_blk/ + cp drivers/block/null_blk//null_blk.ko /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk// + ls -lrth /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko -rw-r--r--. 1 root root 1.1M Jun 29 17:06 /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko + sleep 1 + dmesg -c ++ ./compile_nullb.sh + umount /mnt/nullb0 umount: /mnt/nullb0: not mounted. + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + dmesg -c + modprobe -r null_blk + lsmod + grep null_blk ++ nproc + make -j 48 M=drivers/block modules + HOST=drivers/block/null_blk/ ++ uname -r + HOST_DEST=/lib/modules/5.18.0blk+/kernel/drivers/block/null_blk/ + cp drivers/block/null_blk//null_blk.ko /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk// + ls -lrth /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko -rw-r--r--. 1 root root 1.1M Jun 29 17:06 /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko + sleep 1 + dmesg -c ++ modprobe null_blk nr_devices=0 ++ sleep 1 ++ mkdir config/nullb/nullb0 ++ tree config/nullb/nullb0 config/nullb/nullb0 ├── badblocks ├── blocking ├── blocksize ├── cache_size ├── completion_nsec ├── discard ├── home_node ├── hw_queue_depth ├── index ├── irqmode ├── max_sectors ├── mbps ├── memory_backed ├── poll_queues ├── power ├── queue_mode ├── size ├── submit_queues ├── use_per_node_hctx ├── verify ├── virt_boundary ├── zone_capacity ├── zoned ├── zone_max_active ├── zone_max_open ├── zone_nr_conv └── zone_size 0 directories, 27 files ++ echo 1 ++ echo 512 ++ echo 20480 ++ echo 1 ++ sleep 2 +++ cat config/nullb/nullb0/index ++ IDX=0 ++ lsblk ++ grep null0 ++ sleep 1 ++ mkfs.xfs -f /dev/nullb0 meta-data=/dev/nullb0 isize=512 agcount=4, agsize=1310720 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=5242880, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 ++ mount /dev/nullb0 /mnt/backend/ ++ sleep 1 ++ mount ++ column -t ++ grep nvme ++ dd if=/dev/zero of=/mnt/backend/nvme1n1 count=2621440 bs=4096 2621440+0 records in 2621440+0 records out 10737418240 bytes (11 GB, 10 GiB) copied, 5.01608 s, 2.1 GB/s ++ file /mnt/backend/nvme1n1 /mnt/backend/nvme1n1: data ++ make_target ++ tree /sys/kernel/config /sys/kernel/config ├── nullb │ ├── features │ └── nullb0 │ ├── badblocks │ ├── blocking │ ├── blocksize │ ├── cache_size │ ├── completion_nsec │ ├── discard │ ├── home_node │ ├── hw_queue_depth │ ├── index │ ├── irqmode │ ├── max_sectors │ ├── mbps │ ├── memory_backed │ ├── poll_queues │ ├── power │ ├── queue_mode │ ├── size │ ├── submit_queues │ ├── use_per_node_hctx │ ├── verify │ ├── virt_boundary │ ├── zone_capacity │ ├── zoned │ ├── zone_max_active │ ├── zone_max_open │ ├── zone_nr_conv │ └── zone_size └── nvmet ├── hosts ├── ports └── subsystems 6 directories, 28 files ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/ ++ for i in 1 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn//namespaces/1 ++ echo -n /mnt/backend/nvme1n1 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn//namespaces/1/device_path /mnt/backend/nvme1n1 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn//namespaces/1/buffered_io 0 ++ echo 1 ++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop ++ echo -n 1 ++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn/ /sys/kernel/config/nvmet/ports/1/subsystems/ ++ sleep 1 ++ connect ++ echo transport=loop,nqn=testnqn ++ sleep 1 ++ dmesg -c [ 3436.671070] null_blk: module loaded [ 3437.678812] null_blk: disk nullb0 created [ 3440.700250] XFS (nullb0): Mounting V5 Filesystem [ 3440.701686] XFS (nullb0): Ending clean mount [ 3440.701772] xfs filesystem being mounted at /mnt/backend supports timestamps until 2038 (0x7fffffff) [ 3446.742777] nvmet: adding nsid 1 to subsystem testnqn [ 3447.752282] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:55bf9c5a-2992-4ffc-ac53-18003029a0b9. [ 3447.752423] nvme nvme1: creating 48 I/O queues. [ 3447.758869] nvme nvme1: new ctrl: "testnqn" linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1 linux-block (for-next) # dmesg -c [ 3452.675959] nvme_setup_verify 844 [ 3452.675969] nvmet: nvmet_file_execute_verify 525 [ 3452.675971] nvmet: nvmet_file_emulate_verify_work 502 [ 3452.675972] nvmet: do_direct_io_emulate_verify 431 linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1 NVME Verify Success linux-block (for-next) # dmesg -c [ 3459.794385] nvmet: nvmet_file_execute_verify 525 [ 3459.794389] nvmet: nvmet_file_emulate_verify_work 502 [ 3459.794391] nvmet: do_direct_io_emulate_verify 431 linux-block (for-next) # linux-block (for-next) # linux-block (for-next) # ./delete.sh 1 + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.338s user 0m0.001s sys 0m0.004s ++ shuf -i 1-1 -n 1 + for i in `shuf -i 1-$NN -n $NN` + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rmdir config/nullb/nullb0 + sleep 2 + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + sleep 1 + rmdir /sys/kernel/config/nvmet/ports/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + sleep 1 + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files ################################# NVMe PCIe device ########################### linux-block (for-next) # linux-block (for-next) # linux-block (for-next) # modprobe nvme linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme0n1 linux-block (for-next) # dmesg -c [ 2763.432194] nvme nvme0: pci function 0000:00:04.0 [ 2763.473827] nvme nvme0: 48/0/0 default/read/poll queues [ 2763.478868] nvme nvme0: Ignoring bogus Namespace Identifiers [ 2766.583923] nvme_setup_verify 844
