On 4/23/22 06:57, Elliott Balsley wrote: > I'm testing some NVMe drives and I notice the IO Depth reported by fio > output is lower than what I specify in the job. Is the command-line > output showing the attempted depth or the actual achieved depth? > Here's an example output where I tried iodepth=32. The first line (IO > depths) shows a mix of 8 and 16. The next 2 lines (submit and > complete) show 4. What's the difference between the first and second > lines? I was expecting "submit" to be 32 and "complete" to maybe be > lower based on hardware and OS limitations. Oddly enough, if I > specify iodepth=4 then performance is actually a bit better (32K > instead of 35K) even though the submit/complete depths shown remain > the same. > Also — is there a way to find out what is limiting this? The Micron > 9300 Pro should support QD=512 according to its spec sheet. > > % fio --randrepeat=1 --name=test --bs=4k --iodepth=32 --size=1G > --readwrite=randread --directory /Volumes/Micron9300/elliott > --ioengine posixaio If it is on Linux, try libaio. The file system may be limiting (blocking) IO submission due to some internal IO processing/inode locking. Try testing a similar workload on the raw drive first (--filename=/dev/nvmeXnY) and see what the queue depth is. > test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > 4096B-4096B, ioengine=posixaio, iodepth=32 > fio-3.30 > Starting 1 process > Jobs: 1 (f=1): [r(1)][100.0%][r=137MiB/s][r=35.1k IOPS][eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=6903: Fri Apr 22 12:25:34 2022 > read: IOPS=35.0k, BW=137MiB/s (143MB/s)(1024MiB/7498msec) > slat (nsec): min=0, max=46000, avg=823.84, stdev=784.94 > clat (usec): min=179, max=874, avg=455.01, stdev=50.98 > lat (usec): min=180, max=875, avg=455.84, stdev=50.88 > clat percentiles (usec): > | 1.00th=[ 318], 5.00th=[ 367], 10.00th=[ 392], 20.00th=[ 416], > | 30.00th=[ 433], 40.00th=[ 445], 50.00th=[ 457], 60.00th=[ 469], > | 70.00th=[ 482], 80.00th=[ 494], 90.00th=[ 515], 95.00th=[ 537], > | 99.00th=[ 578], 99.50th=[ 594], 99.90th=[ 652], 99.95th=[ 668], > | 99.99th=[ 717] > bw ( KiB/s): min=139419, max=140595, per=100.00%, avg=140110.71, > stdev=340.75, samples=14 > iops : min=34854, max=35148, avg=35027.43, stdev=85.14, samples=14 > lat (usec) : 250=0.06%, 500=82.67%, 750=17.27%, 1000=0.01% > cpu : usr=19.97%, sys=13.74%, ctx=195959, majf=0, minf=28 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=43.4%, 16=56.6%, 32=0.0%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=99.1%, 8=0.9%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% > issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), > io=1024MiB (1074MB), run=7498-7498msec > > Thanks, > Elliott -- Damien Le Moal Western Digital Research
