I'm testing some NVMe drives and I notice the IO Depth reported by fio
output is lower than what I specify in the job. Is the command-line
output showing the attempted depth or the actual achieved depth?
Here's an example output where I tried iodepth=32. The first line (IO
depths) shows a mix of 8 and 16. The next 2 lines (submit and
complete) show 4. What's the difference between the first and second
lines? I was expecting "submit" to be 32 and "complete" to maybe be
lower based on hardware and OS limitations. Oddly enough, if I
specify iodepth=4 then performance is actually a bit better (32K
instead of 35K) even though the submit/complete depths shown remain
the same.
Also — is there a way to find out what is limiting this? The Micron
9300 Pro should support QD=512 according to its spec sheet.
% fio --randrepeat=1 --name=test --bs=4k --iodepth=32 --size=1G
--readwrite=randread --directory /Volumes/Micron9300/elliott
--ioengine posixaio
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=posixaio, iodepth=32
fio-3.30
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=137MiB/s][r=35.1k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=6903: Fri Apr 22 12:25:34 2022
read: IOPS=35.0k, BW=137MiB/s (143MB/s)(1024MiB/7498msec)
slat (nsec): min=0, max=46000, avg=823.84, stdev=784.94
clat (usec): min=179, max=874, avg=455.01, stdev=50.98
lat (usec): min=180, max=875, avg=455.84, stdev=50.88
clat percentiles (usec):
| 1.00th=[ 318], 5.00th=[ 367], 10.00th=[ 392], 20.00th=[ 416],
| 30.00th=[ 433], 40.00th=[ 445], 50.00th=[ 457], 60.00th=[ 469],
| 70.00th=[ 482], 80.00th=[ 494], 90.00th=[ 515], 95.00th=[ 537],
| 99.00th=[ 578], 99.50th=[ 594], 99.90th=[ 652], 99.95th=[ 668],
| 99.99th=[ 717]
bw ( KiB/s): min=139419, max=140595, per=100.00%, avg=140110.71,
stdev=340.75, samples=14
iops : min=34854, max=35148, avg=35027.43, stdev=85.14, samples=14
lat (usec) : 250=0.06%, 500=82.67%, 750=17.27%, 1000=0.01%
cpu : usr=19.97%, sys=13.74%, ctx=195959, majf=0, minf=28
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=43.4%, 16=56.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.9%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s),
io=1024MiB (1074MB), run=7498-7498msec
Thanks,
Elliott
