hey Jeff,
Good catch! I have run the following command:
fio --name=seqread --numjobs=1 --time_based --runtime=60s --ramp_time=2s
--iodepth=8 --ioengine=libaio --direct=1 --verify=0 --group_reporting=1
--bs=1M --rw=read --size=1G --filename=/dev/sda
(/sda and /sdd are the drives I have on one of the pools), and this is
what I get:
seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB,
(T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=388MiB/s][r=388 IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=2368687: Thu Apr 4 20:56:06
2024
read: IOPS=382, BW=383MiB/s (401MB/s)(22.4GiB/60020msec)
slat (usec): min=17, max=3098, avg=68.94, stdev=46.04
clat (msec): min=14, max=367, avg=20.84, stdev= 6.61
lat (msec): min=15, max=367, avg=20.91, stdev= 6.61
clat percentiles (msec):
| 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[
21],
| 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[
21],
| 70.00th=[ 21], 80.00th=[ 21], 90.00th=[ 21], 95.00th=[
21],
| 99.00th=[ 25], 99.50th=[ 31], 99.90th=[ 48], 99.95th=[
50],
| 99.99th=[ 368]
bw ( KiB/s): min=215040, max=399360, per=100.00%, avg=392047.06,
stdev=19902.89, samples=120
iops : min= 210, max= 390, avg=382.55, stdev=19.43,
samples=120
lat (msec) : 20=0.19%, 50=99.80%, 100=0.01%, 500=0.03%
cpu : usr=0.39%, sys=1.93%, ctx=45947, majf=0, minf=37
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwts: total=22954,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=383MiB/s (401MB/s), 383MiB/s-383MiB/s (401MB/s-401MB/s),
io=22.4GiB (24.1GB), run=60020-60020msec
Disk stats (read/write):
sda: ios=23817/315, merge=0/0, ticks=549704/132687, in_queue=683613,
util=99.93%
400 MBps!!! This is a number I have never experienced. I understand this
means I need to go back to the openzfs chat/forum?
---
Felix Rubio
"Don't believe what you're told. Double check."
