Hi, I use fio 3.19 to test NvmeSSD,cmd string is: -------------------------------------------------------------------------------- fio --filename=/dev/nvme0n1 --rw=randwrite --offset=0 --size=10% --bs=128k --iodepth=64 --numjobs=1 --ioengine=libaio --name=randwrite_data --allow_file_create=0 --thread --direct=1 --group_reporting=1 --verify_dump=1 --verify_backlog=1 --randrepeat=0 --do_verify=1 --verify=pattern --verify_pattern=0x51d4b0e0 --bssplit=512:4k:16k:32k:64k:128k:1024k -------------------------------------------------------------------------------- Then a verify error occurs, -------------------------------------------------------------------------------- randwrite_data: (g=0): rw=randwrite, bs=(R) 512B-1024KiB, (W) 512B-1024KiB, (T) 512B-1024KiB, ioengine=libaio, iodepth=64 fio-3.19 Starting 1 thread fio: got pattern '00', wanted '51'. Bad bits 3 fio: bad pattern block offset 384 pattern: verify failed at file /dev/nvme0n1 offset 70523721971, length 3769685073 (requested block: offset=68822685696, length=1048576) received data dumped as nvme0n1.69899573248.received expected data dumped as nvme0n1.69899573248.expected fio: verify type mismatch (57520 media, 18 given) randwrite_data: (groupid=0, jobs=1): err= 0: pid=2990: Thu Sep 1 14:10:23 2022 read: IOPS=167M, BW=8943GiB/s (9602GB/s)(89.4GiB/10msec) slat (nsec): min=559, max=21279, avg=1135.63, stdev=853.10 clat (usec): min=13, max=51046, avg=3229.89, stdev=3195.51 lat (usec): min=14, max=51053, avg=3231.04, stdev=3196.01 clat percentiles (usec): | 1.00th=[ 217], 5.00th=[ 545], 10.00th=[ 857], 20.00th=[ 1369], | 30.00th=[ 1795], 40.00th=[ 2147], 50.00th=[ 2474], 60.00th=[ 2802], | 70.00th=[ 3228], 80.00th=[ 4080], 90.00th=[ 6128], 95.00th=[ 8848], | 99.00th=[17171], 99.50th=[21627], 99.90th=[30278], 99.95th=[33162], | 99.99th=[38536] write: IOPS=12.7k, BW=698MiB/s (732MB/s)(89.4GiB/131206msec); 0 zone resets slat (nsec): min=639, max=36626, avg=1299.14, stdev=901.82 clat (usec): min=8, max=31311, avg=1797.57, stdev=1917.38 lat (usec): min=9, max=31312, avg=1798.89, stdev=1917.58 clat percentiles (usec): | 1.00th=[ 16], 5.00th=[ 27], 10.00th=[ 42], 20.00th=[ 108], | 30.00th=[ 334], 40.00th=[ 824], 50.00th=[ 1369], 60.00th=[ 1909], | 70.00th=[ 2409], 80.00th=[ 3064], 90.00th=[ 4146], 95.00th=[ 5276], | 99.00th=[ 8586], 99.50th=[10159], 99.90th=[13960], 99.95th=[15533], | 99.99th=[18744] bw ( KiB/s): min=497348, max=1063791, per=100.00%, avg=715944.11, stdev=115634.92, samples=261 iops : min= 4238, max=18726, avg=12722.05, stdev=4574.05, samples=261 lat (usec) : 10=0.01%, 20=1.23%, 50=4.85%, 100=3.68%, 250=4.70% lat (usec) : 500=4.63%, 750=4.35%, 1000=4.44% lat (msec) : 2=20.84%, 4=35.50%, 10=13.53%, 20=1.91%, 50=0.32% lat (msec) : 100=0.01% cpu : usr=5.62%, sys=3.29%, ctx=2799800, majf=0, minf=266 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1668079,1668079,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=8943GiB/s (9602GB/s), 8943GiB/s-8943GiB/s (9602GB/s-9602GB/s), io=89.4GiB (96.0GB), run=10-10msec WRITE: bw=698MiB/s (732MB/s), 698MiB/s-698MiB/s (732MB/s-732MB/s), io=89.4GiB (96.0GB), run=131206-131206msec Disk stats (read/write): nvme0n1: ios=1772035/1772257, merge=0/0, ticks=6269444/3172530, in_queue=9441974, util=100.00% -------------------------------------------------------------------------------- fio: got pattern '00', wanted '51'. Bad bits 3 fio: bad pattern block offset 384 pattern: verify failed at file /dev/nvme0n1 offset 70523721971, length 3769685073 (requested block: offset=68822685696, length=1048576) received data dumped as nvme0n1.69899573248.received expected data dumped as nvme0n1.69899573248.expected fio: verify type mismatch (57520 media, 18 given) "offset 384", "offset 70523721971, length 3769685073", "offset=68822685696, length=1048576", "nvme0n1.69899573248.received" These number has no any relation, like equal or sum equl. Maybe this is an error or mistake??