On Mon, Oct 11, 2021 at 09:15:47AM -0600, Jens Axboe wrote:
> On 10/11/21 7:44 AM, Luís Henriques wrote:
> > On Mon, Oct 11, 2021 at 06:58:52AM -0600, Jens Axboe wrote:
> >> On 10/11/21 4:27 AM, Luís Henriques wrote:
> >>> While running fstests generic/095 against ext4 on a zram device I started
> >>> seeing fio crashing. Fix it by making sure io_u->file isn't NULL before
> >>> accessing it.
> >>>
> >>> Signed-off-by: Luís Henriques <lhenriques@xxxxxxx>
> >>> ---
> >>> io_u.c | 3 ++-
> >>> 1 file changed, 2 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/io_u.c b/io_u.c
> >>> index 5289b5d1d9c6..b8e715d4118c 100644
> >>> --- a/io_u.c
> >>> +++ b/io_u.c
> >>> @@ -2009,7 +2009,8 @@ static void io_completed(struct thread_data *td, struct io_u **io_u_ptr,
> >>> io_u->xfer_buf += bytes;
> >>> io_u->offset += bytes;
> >>> td->ts.short_io_u[io_u->ddir]++;
> >>> - if (io_u->offset < io_u->file->real_file_size) {
> >>> + if (io_u->file &&
> >>> + (io_u->offset < io_u->file->real_file_size)) {
> >>> requeue_io_u(td, io_u_ptr);
> >>> return;
> >>> }
> >>
> >> This will prevent the crash, but I'm wondering why io_u-> == NULL for this case.
> >> It really should be a valid file.
> >>
> >> Can you let me know exactly how you're reproducing this? Then I'll give it
> >> a whirl too.
> >
> > Sure, here's my recipe for reproducing this bug:
> >
> > # modprobe zram num_devices=2
> > # echo 1G > /sys/block/zram0/disksize
> > # echo 1G > /sys/block/zram1/disksize
> > # mkfs.ext4 /dev/zram0
> > # mkfs.ext4 /dev/zram1
> >
> > # ./check generic/095
>
> I know you fixed the issue in the test by now, but any chance you can try
> this one and see if it correctly reports full residual instead of trying
> to requeue and crash?
Sure! So, I patched fio with this change and executed the generic/095
test twice: one run without the test fix, another with the test fix. I
can confirm that fio didn't crash in any of the runs, but I'm not sure how
to check if it's correctly reporting residual.
I'm attaching the test logs (for the failure run) in case that helps.
Cheers,
--
Luís
>
> diff --git a/io_u.c b/io_u.c
> index 5289b5d1d9c6..586a4befdce0 100644
> --- a/io_u.c
> +++ b/io_u.c
> @@ -2004,7 +2004,7 @@ static void io_completed(struct thread_data *td, struct io_u **io_u_ptr,
> * Make sure we notice short IO from here, and requeue them
> * appropriately!
> */
> - if (io_u->resid) {
> + if (bytes && io_u->resid) {
> io_u->xfer_buflen = io_u->resid;
> io_u->xfer_buf += bytes;
> io_u->offset += bytes;
>
> --
> Jens Axboe
>
fio --bs=8k --iodepth=16 --iodepth_batch=8 --randrepeat=1 --size=1m --directory=/tmp/mnt/scratch --numjobs=5 --name=job1 --ioengine=sync --bs=1k --direct=1 --rw=randread --filename=file1:file2 --name=job2 --ioengine=libaio --rw=randwrite --direct=1 --filename=file1:file2 --name=job3 --bs=1k --ioengine=posixaio --rw=randwrite --direct=1 --filename=file1:file2 --name=job4 --ioengine=splice --direct=1 --rw=randwrite --filename=file1:file2 --name=job5 --bs=1k --ioengine=sync --rw=randread --filename=file1:file2 --name=job6 --ioengine=posixaio --rw=randwrite --filename=file1:file2 --name=job7 --ioengine=splice --rw=randwrite --filename=file1:file2 --name=job8 --ioengine=mmap --rw=randwrite --bs=1k --filename=file1:file2 --name=job9 --ioengine=mmap --rw=randwrite --direct=1 --bs=4096 --filename=file1:file2
Discarding device blocks: 0/262144 done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: e77d8742-2325-4f0c-a67c-9900cb195663
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: 0/8 done
Writing inode tables: 0/8 done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: 0/8 done
job1: (g=0): rw=randread, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=sync, iodepth=16
...
job2: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
...
job3: (g=0): rw=randwrite, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=posixaio, iodepth=16
...
job4: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=splice, iodepth=16
...
job5: (g=0): rw=randread, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=sync, iodepth=16
...
job6: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=posixaio, iodepth=16
...
job7: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=splice, iodepth=16
...
job8: (g=0): rw=randwrite, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=mmap, iodepth=16
...
job9: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=mmap, iodepth=16
...
fio-3.28-57-g63b8-dirty
Starting 45 processes
job1: Laying out IO files (2 files / total 1MiB)
fio: pid=2364, got signal=6
fio: pid=2367, got signal=6
fio: pid=2368, got signal=6
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2354: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=2, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2355: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=3, majf=3, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2356: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=6, majf=6, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2357: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2358: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=8, majf=8, minf=24
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2359: Mon Oct 11 16:33:05 2021
write: IOPS=1600, BW=12.5MiB/s (13.1MB/s)(1024KiB/80msec); 0 zone resets
slat (usec): min=25, max=37322, avg=2684.09, stdev=7381.40
clat (usec): min=4, max=12289, avg=3860.08, stdev=3161.06
lat (usec): min=77, max=47504, avg=6544.27, stdev=9652.17
clat percentiles (usec):
| 1.00th=[ 82], 5.00th=[ 652], 10.00th=[ 1221], 20.00th=[ 1598],
| 30.00th=[ 1713], 40.00th=[ 1778], 50.00th=[ 2343], 60.00th=[ 3359],
| 70.00th=[ 3949], 80.00th=[ 6980], 90.00th=[ 8586], 95.00th=[10814],
| 99.00th=[11994], 99.50th=[12256], 99.90th=[12256], 99.95th=[12256],
| 99.99th=[12256]
lat (usec) : 10=0.78%, 100=0.78%, 250=0.78%, 500=0.78%, 750=3.12%
lat (usec) : 1000=2.34%
lat (msec) : 2=38.28%, 4=24.22%, 10=21.88%, 20=7.03%
cpu : usr=0.00%, sys=13.92%, ctx=61, majf=3, minf=19
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2360: Mon Oct 11 16:33:05 2021
write: IOPS=2206, BW=17.2MiB/s (18.1MB/s)(1024KiB/58msec); 0 zone resets
slat (usec): min=19, max=22227, avg=1293.17, stdev=4243.34
clat (nsec): min=1744, max=13937k, avg=2624525.45, stdev=3543041.64
lat (usec): min=55, max=22739, avg=3917.79, stdev=5242.31
clat percentiles (usec):
| 1.00th=[ 57], 5.00th=[ 233], 10.00th=[ 437], 20.00th=[ 545],
| 30.00th=[ 570], 40.00th=[ 594], 50.00th=[ 627], 60.00th=[ 889],
| 70.00th=[ 2671], 80.00th=[ 3556], 90.00th=[ 9503], 95.00th=[11469],
| 99.00th=[12649], 99.50th=[13960], 99.90th=[13960], 99.95th=[13960],
| 99.99th=[13960]
lat (usec) : 2=0.78%, 100=0.78%, 250=3.91%, 500=5.47%, 750=46.88%
lat (usec) : 1000=3.91%
lat (msec) : 2=4.69%, 4=14.06%, 10=12.50%, 20=7.03%
cpu : usr=8.77%, sys=0.00%, ctx=33, majf=1, minf=21
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2361: Mon Oct 11 16:33:05 2021
write: IOPS=1662, BW=13.0MiB/s (13.6MB/s)(1024KiB/77msec); 0 zone resets
slat (usec): min=21, max=31476, avg=2285.17, stdev=6451.78
clat (nsec): min=1461, max=8546.3k, avg=3394454.86, stdev=2584376.00
lat (usec): min=30, max=38816, avg=5679.71, stdev=7944.51
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 243], 10.00th=[ 938], 20.00th=[ 1336],
| 30.00th=[ 1762], 40.00th=[ 2114], 50.00th=[ 2278], 60.00th=[ 2573],
| 70.00th=[ 5080], 80.00th=[ 5932], 90.00th=[ 7963], 95.00th=[ 8291],
| 99.00th=[ 8586], 99.50th=[ 8586], 99.90th=[ 8586], 99.95th=[ 8586],
| 99.99th=[ 8586]
lat (usec) : 2=0.78%, 50=0.78%, 100=1.56%, 250=2.34%, 500=1.56%
lat (usec) : 1000=3.91%
lat (msec) : 2=24.22%, 4=34.38%, 10=30.47%
cpu : usr=0.00%, sys=6.58%, ctx=77, majf=1, minf=21
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2362: Mon Oct 11 16:33:05 2021
write: IOPS=1684, BW=13.2MiB/s (13.8MB/s)(1024KiB/76msec); 0 zone resets
slat (usec): min=18, max=34986, avg=2547.30, stdev=6865.46
clat (nsec): min=1454, max=12139k, avg=3714736.58, stdev=3282289.17
lat (usec): min=26, max=45556, avg=6262.11, stdev=9137.41
clat percentiles (usec):
| 1.00th=[ 28], 5.00th=[ 190], 10.00th=[ 343], 20.00th=[ 465],
| 30.00th=[ 1942], 40.00th=[ 2245], 50.00th=[ 2409], 60.00th=[ 2802],
| 70.00th=[ 3916], 80.00th=[ 6783], 90.00th=[ 8455], 95.00th=[11207],
| 99.00th=[11994], 99.50th=[12125], 99.90th=[12125], 99.95th=[12125],
| 99.99th=[12125]
lat (usec) : 2=0.78%, 50=0.78%, 100=0.78%, 250=4.69%, 500=13.28%
lat (msec) : 2=13.28%, 4=36.72%, 10=22.66%, 20=7.03%
cpu : usr=0.00%, sys=5.33%, ctx=90, majf=3, minf=19
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2363: Mon Oct 11 16:33:05 2021
write: IOPS=1662, BW=13.0MiB/s (13.6MB/s)(1024KiB/77msec); 0 zone resets
slat (usec): min=20, max=36554, avg=2615.40, stdev=7062.59
clat (nsec): min=1561, max=13205k, avg=3852028.32, stdev=3205515.13
lat (usec): min=36, max=47273, avg=6467.50, stdev=9399.31
clat percentiles (usec):
| 1.00th=[ 38], 5.00th=[ 343], 10.00th=[ 1020], 20.00th=[ 1500],
| 30.00th=[ 1663], 40.00th=[ 1975], 50.00th=[ 2769], 60.00th=[ 3359],
| 70.00th=[ 4424], 80.00th=[ 6259], 90.00th=[ 8160], 95.00th=[11207],
| 99.00th=[13173], 99.50th=[13173], 99.90th=[13173], 99.95th=[13173],
| 99.99th=[13173]
lat (usec) : 2=0.78%, 50=0.78%, 100=0.78%, 250=1.56%, 500=3.12%
lat (usec) : 750=0.78%, 1000=1.56%
lat (msec) : 2=31.25%, 4=28.91%, 10=23.44%, 20=7.03%
cpu : usr=0.00%, sys=6.58%, ctx=81, majf=1, minf=21
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2364: Mon Oct 11 16:33:05 2021
lat (msec) : 10=18.75%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.3%, 4=84.2%, 8=5.3%, 16=5.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2365: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=2, majf=1, minf=27
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.6%, 4=94.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2366: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=5, majf=3, minf=27
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.6%, 4=94.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2367: Mon Oct 11 16:33:05 2021
lat (usec) : 1000=18.75%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.3%, 4=84.2%, 8=5.3%, 16=5.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2368: Mon Oct 11 16:33:05 2021
lat (msec) : 4=18.75%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.3%, 4=84.2%, 8=5.3%, 16=5.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2369: Mon Oct 11 16:33:05 2021
write: IOPS=1641, BW=12.8MiB/s (13.4MB/s)(1024KiB/78msec); 0 zone resets
clat (usec): min=32, max=18196, avg=511.89, stdev=1705.10
lat (usec): min=32, max=18196, avg=512.09, stdev=1705.10
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 36], 10.00th=[ 40], 20.00th=[ 61],
| 30.00th=[ 73], 40.00th=[ 88], 50.00th=[ 149], 60.00th=[ 176],
| 70.00th=[ 285], 80.00th=[ 494], 90.00th=[ 955], 95.00th=[ 2245],
| 99.00th=[ 3851], 99.50th=[18220], 99.90th=[18220], 99.95th=[18220],
| 99.99th=[18220]
lat (usec) : 50=14.06%, 100=28.91%, 250=25.00%, 500=12.50%, 750=7.81%
lat (usec) : 1000=3.12%
lat (msec) : 2=3.12%, 4=4.69%, 20=0.78%
cpu : usr=0.00%, sys=12.99%, ctx=83, majf=5, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2370: Mon Oct 11 16:33:05 2021
write: IOPS=2064, BW=16.1MiB/s (16.9MB/s)(1024KiB/62msec); 0 zone resets
clat (usec): min=25, max=16986, avg=470.55, stdev=1703.89
lat (usec): min=25, max=16986, avg=470.70, stdev=1703.90
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 33], 10.00th=[ 34], 20.00th=[ 38],
| 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 55],
| 70.00th=[ 76], 80.00th=[ 363], 90.00th=[ 947], 95.00th=[ 2147],
| 99.00th=[ 5604], 99.50th=[16909], 99.90th=[16909], 99.95th=[16909],
| 99.99th=[16909]
lat (usec) : 50=56.25%, 100=16.41%, 250=6.25%, 500=8.59%, 750=0.78%
lat (usec) : 1000=2.34%
lat (msec) : 2=3.12%, 4=3.91%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=9.84%, ctx=45, majf=2, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2371: Mon Oct 11 16:33:05 2021
write: IOPS=1729, BW=13.5MiB/s (14.2MB/s)(1024KiB/74msec); 0 zone resets
clat (usec): min=30, max=18586, avg=476.04, stdev=1774.05
lat (usec): min=31, max=18586, avg=476.26, stdev=1774.05
clat percentiles (usec):
| 1.00th=[ 32], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 34],
| 30.00th=[ 56], 40.00th=[ 89], 50.00th=[ 93], 60.00th=[ 163],
| 70.00th=[ 206], 80.00th=[ 343], 90.00th=[ 840], 95.00th=[ 1614],
| 99.00th=[ 5407], 99.50th=[18482], 99.90th=[18482], 99.95th=[18482],
| 99.99th=[18482]
lat (usec) : 50=25.00%, 100=27.34%, 250=20.31%, 500=14.06%, 750=3.12%
lat (usec) : 1000=2.34%
lat (msec) : 2=3.12%, 4=2.34%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=13.70%, ctx=49, majf=2, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2372: Mon Oct 11 16:33:05 2021
write: IOPS=1706, BW=13.3MiB/s (14.0MB/s)(1024KiB/75msec); 0 zone resets
clat (usec): min=32, max=17596, avg=565.07, stdev=1719.54
lat (usec): min=32, max=17596, avg=565.27, stdev=1719.57
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 67],
| 30.00th=[ 78], 40.00th=[ 104], 50.00th=[ 130], 60.00th=[ 180],
| 70.00th=[ 289], 80.00th=[ 379], 90.00th=[ 1303], 95.00th=[ 2966],
| 99.00th=[ 4228], 99.50th=[17695], 99.90th=[17695], 99.95th=[17695],
| 99.99th=[17695]
lat (usec) : 50=15.62%, 100=23.44%, 250=26.56%, 500=16.41%, 750=3.91%
lat (usec) : 1000=2.34%
lat (msec) : 2=5.47%, 4=3.91%, 10=1.56%, 20=0.78%
cpu : usr=6.76%, sys=8.11%, ctx=102, majf=0, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2373: Mon Oct 11 16:33:05 2021
write: IOPS=1620, BW=12.7MiB/s (13.3MB/s)(1024KiB/79msec); 0 zone resets
clat (usec): min=35, max=6962, avg=520.71, stdev=1210.46
lat (usec): min=35, max=6962, avg=520.98, stdev=1210.49
clat percentiles (usec):
| 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 65], 20.00th=[ 86],
| 30.00th=[ 100], 40.00th=[ 118], 50.00th=[ 151], 60.00th=[ 172],
| 70.00th=[ 221], 80.00th=[ 326], 90.00th=[ 1156], 95.00th=[ 3163],
| 99.00th=[ 6849], 99.50th=[ 6980], 99.90th=[ 6980], 99.95th=[ 6980],
| 99.99th=[ 6980]
lat (usec) : 50=8.59%, 100=20.31%, 250=45.31%, 500=12.50%, 750=1.56%
lat (usec) : 1000=1.56%
lat (msec) : 2=2.34%, 4=4.69%, 10=3.12%
cpu : usr=0.00%, sys=19.23%, ctx=113, majf=6, minf=15
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2374: Mon Oct 11 16:33:05 2021
read: IOPS=85.3k, BW=83.3MiB/s (87.4MB/s)(1024KiB/12msec)
clat (nsec): min=582, max=104452, avg=965.48, stdev=3594.28
lat (nsec): min=622, max=104492, avg=1006.11, stdev=3595.08
clat percentiles (nsec):
| 1.00th=[ 588], 5.00th=[ 604], 10.00th=[ 620], 20.00th=[ 652],
| 30.00th=[ 684], 40.00th=[ 700], 50.00th=[ 716], 60.00th=[ 732],
| 70.00th=[ 748], 80.00th=[ 772], 90.00th=[ 812], 95.00th=[ 900],
| 99.00th=[ 4048], 99.50th=[ 5280], 99.90th=[ 46336], 99.95th=[104960],
| 99.99th=[104960]
lat (nsec) : 750=69.43%, 1000=26.46%
lat (usec) : 2=0.68%, 4=2.34%, 10=0.78%, 20=0.10%, 50=0.10%
lat (usec) : 250=0.10%
cpu : usr=9.09%, sys=0.00%, ctx=9, majf=6, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2375: Mon Oct 11 16:33:05 2021
read: IOPS=73.1k, BW=71.4MiB/s (74.9MB/s)(1024KiB/14msec)
clat (nsec): min=562, max=17276, avg=759.74, stdev=815.84
lat (nsec): min=603, max=17596, avg=799.78, stdev=822.30
clat percentiles (nsec):
| 1.00th=[ 588], 5.00th=[ 596], 10.00th=[ 604], 20.00th=[ 620],
| 30.00th=[ 644], 40.00th=[ 660], 50.00th=[ 684], 60.00th=[ 700],
| 70.00th=[ 724], 80.00th=[ 748], 90.00th=[ 788], 95.00th=[ 820],
| 99.00th=[ 2768], 99.50th=[ 4896], 99.90th=[16064], 99.95th=[17280],
| 99.99th=[17280]
lat (nsec) : 750=81.25%, 1000=16.70%
lat (usec) : 2=0.68%, 4=0.68%, 10=0.49%, 20=0.20%
cpu : usr=0.00%, sys=7.69%, ctx=4, majf=3, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2376: Mon Oct 11 16:33:05 2021
read: IOPS=93.1k, BW=90.9MiB/s (95.3MB/s)(1024KiB/11msec)
clat (nsec): min=594, max=14728, avg=1045.56, stdev=1284.62
lat (nsec): min=630, max=14769, avg=1086.76, stdev=1293.48
clat percentiles (nsec):
| 1.00th=[ 604], 5.00th=[ 620], 10.00th=[ 636], 20.00th=[ 660],
| 30.00th=[ 684], 40.00th=[ 700], 50.00th=[ 716], 60.00th=[ 732],
| 70.00th=[ 756], 80.00th=[ 788], 90.00th=[ 1064], 95.00th=[ 3568],
| 99.00th=[ 5664], 99.50th=[ 8256], 99.90th=[14144], 99.95th=[14784],
| 99.99th=[14784]
lat (nsec) : 750=67.48%, 1000=22.27%
lat (usec) : 2=1.95%, 4=5.76%, 10=2.15%, 20=0.39%
cpu : usr=10.00%, sys=0.00%, ctx=6, majf=5, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2377: Mon Oct 11 16:33:05 2021
read: IOPS=85.3k, BW=83.3MiB/s (87.4MB/s)(1024KiB/12msec)
clat (nsec): min=641, max=16976, avg=878.51, stdev=857.12
lat (nsec): min=681, max=17021, avg=923.44, stdev=864.64
clat percentiles (nsec):
| 1.00th=[ 652], 5.00th=[ 668], 10.00th=[ 684], 20.00th=[ 700],
| 30.00th=[ 724], 40.00th=[ 748], 50.00th=[ 772], 60.00th=[ 788],
| 70.00th=[ 804], 80.00th=[ 828], 90.00th=[ 868], 95.00th=[ 908],
| 99.00th=[ 4320], 99.50th=[ 5024], 99.90th=[14784], 99.95th=[17024],
| 99.99th=[17024]
lat (nsec) : 750=39.75%, 1000=57.13%
lat (usec) : 2=0.49%, 4=1.37%, 10=1.07%, 20=0.20%
cpu : usr=9.09%, sys=0.00%, ctx=9, majf=8, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2378: Mon Oct 11 16:33:05 2021
read: IOPS=68.3k, BW=66.7MiB/s (69.9MB/s)(1024KiB/15msec)
clat (nsec): min=642, max=27395, avg=978.34, stdev=1424.98
lat (nsec): min=679, max=27933, avg=1023.85, stdev=1434.63
clat percentiles (nsec):
| 1.00th=[ 652], 5.00th=[ 668], 10.00th=[ 676], 20.00th=[ 700],
| 30.00th=[ 732], 40.00th=[ 756], 50.00th=[ 772], 60.00th=[ 796],
| 70.00th=[ 812], 80.00th=[ 844], 90.00th=[ 892], 95.00th=[ 1096],
| 99.00th=[ 6048], 99.50th=[ 8640], 99.90th=[19328], 99.95th=[27520],
| 99.99th=[27520]
lat (nsec) : 750=37.30%, 1000=57.23%
lat (usec) : 2=2.15%, 4=1.07%, 10=1.86%, 20=0.29%, 50=0.10%
cpu : usr=7.14%, sys=0.00%, ctx=9, majf=8, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2379: Mon Oct 11 16:33:05 2021
write: IOPS=1855, BW=14.5MiB/s (15.2MB/s)(1024KiB/69msec); 0 zone resets
slat (nsec): min=141, max=75144, avg=1565.61, stdev=7509.80
clat (usec): min=8, max=33482, avg=6816.01, stdev=8391.68
lat (usec): min=9, max=33483, avg=6817.58, stdev=8392.09
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 19], 10.00th=[ 38], 20.00th=[ 147],
| 30.00th=[ 1467], 40.00th=[ 2180], 50.00th=[ 3556], 60.00th=[ 5014],
| 70.00th=[ 7504], 80.00th=[13435], 90.00th=[17695], 95.00th=[28967],
| 99.00th=[33162], 99.50th=[33424], 99.90th=[33424], 99.95th=[33424],
| 99.99th=[33424]
lat (usec) : 10=3.12%, 20=2.34%, 50=5.47%, 100=7.03%, 250=3.91%
lat (usec) : 500=1.56%, 750=1.56%, 1000=0.78%
lat (msec) : 2=12.50%, 4=14.84%, 10=25.00%, 20=12.50%, 50=9.38%
cpu : usr=0.00%, sys=0.00%, ctx=74, majf=2, minf=27
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=62.5%, 16=37.5%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=96.1%, 8=3.4%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2380: Mon Oct 11 16:33:05 2021
write: IOPS=2560, BW=20.0MiB/s (21.0MB/s)(1024KiB/50msec); 0 zone resets
slat (nsec): min=203, max=148143, avg=2370.61, stdev=13430.86
clat (usec): min=6, max=25433, avg=4766.72, stdev=7242.47
lat (usec): min=6, max=25434, avg=4769.09, stdev=7244.42
clat percentiles (usec):
| 1.00th=[ 7], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 40],
| 30.00th=[ 235], 40.00th=[ 1045], 50.00th=[ 1385], 60.00th=[ 2966],
| 70.00th=[ 4555], 80.00th=[ 6718], 90.00th=[22414], 95.00th=[22938],
| 99.00th=[23462], 99.50th=[25560], 99.90th=[25560], 99.95th=[25560],
| 99.99th=[25560]
lat (usec) : 10=2.34%, 20=4.69%, 50=16.41%, 100=2.34%, 250=4.69%
lat (usec) : 500=1.56%, 750=0.78%, 1000=4.69%
lat (msec) : 2=15.62%, 4=10.16%, 10=23.44%, 20=1.56%, 50=11.72%
cpu : usr=2.04%, sys=0.00%, ctx=41, majf=1, minf=28
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=43.8%, 16=56.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.5%, 8=3.4%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2381: Mon Oct 11 16:33:05 2021
write: IOPS=1910, BW=14.9MiB/s (15.7MB/s)(1024KiB/67msec); 0 zone resets
slat (nsec): min=145, max=63825, avg=1455.53, stdev=6646.46
clat (usec): min=7, max=26472, avg=6863.34, stdev=7084.59
lat (usec): min=8, max=26473, avg=6864.79, stdev=7085.53
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 23], 10.00th=[ 46], 20.00th=[ 285],
| 30.00th=[ 1287], 40.00th=[ 2278], 50.00th=[ 4686], 60.00th=[ 6783],
| 70.00th=[11469], 80.00th=[13435], 90.00th=[17433], 95.00th=[19792],
| 99.00th=[24511], 99.50th=[26346], 99.90th=[26346], 99.95th=[26346],
| 99.99th=[26346]
lat (usec) : 10=1.56%, 20=3.12%, 50=7.03%, 100=5.47%, 250=1.56%
lat (usec) : 500=3.12%, 750=3.91%, 1000=3.12%
lat (msec) : 2=7.81%, 4=11.72%, 10=19.53%, 20=27.34%, 50=4.69%
cpu : usr=0.00%, sys=0.00%, ctx=135, majf=5, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.4%, 8=1.1%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2382: Mon Oct 11 16:33:05 2021
write: IOPS=4266, BW=33.3MiB/s (35.0MB/s)(1024KiB/30msec); 0 zone resets
slat (nsec): min=133, max=106117, avg=2044.11, stdev=9968.18
clat (usec): min=7, max=11660, avg=3046.86, stdev=3452.44
lat (usec): min=7, max=11662, avg=3048.91, stdev=3452.70
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 22], 10.00th=[ 34], 20.00th=[ 91],
| 30.00th=[ 619], 40.00th=[ 930], 50.00th=[ 2057], 60.00th=[ 2737],
| 70.00th=[ 3851], 80.00th=[ 5407], 90.00th=[10814], 95.00th=[11600],
| 99.00th=[11600], 99.50th=[11600], 99.90th=[11600], 99.95th=[11600],
| 99.99th=[11600]
lat (usec) : 10=1.56%, 20=2.34%, 50=10.94%, 100=5.47%, 250=7.03%
lat (usec) : 750=11.72%, 1000=1.56%
lat (msec) : 2=7.81%, 4=26.56%, 10=14.06%, 20=10.94%
cpu : usr=0.00%, sys=0.00%, ctx=37, majf=3, minf=26
IO depths : 1=0.0%, 2=6.2%, 4=6.2%, 8=75.0%, 16=12.5%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.5%, 8=3.9%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2383: Mon Oct 11 16:33:05 2021
write: IOPS=3200, BW=25.0MiB/s (26.2MB/s)(1024KiB/40msec); 0 zone resets
slat (nsec): min=244, max=66266, avg=2227.13, stdev=7105.77
clat (usec): min=7, max=24308, avg=3503.31, stdev=6587.20
lat (usec): min=11, max=24309, avg=3505.53, stdev=6587.27
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 19], 10.00th=[ 34], 20.00th=[ 139],
| 30.00th=[ 155], 40.00th=[ 273], 50.00th=[ 725], 60.00th=[ 889],
| 70.00th=[ 1385], 80.00th=[ 5473], 90.00th=[18482], 95.00th=[19006],
| 99.00th=[24249], 99.50th=[24249], 99.90th=[24249], 99.95th=[24249],
| 99.99th=[24249]
lat (usec) : 10=0.78%, 20=4.69%, 50=7.03%, 100=5.47%, 250=21.88%
lat (usec) : 500=4.69%, 750=10.94%, 1000=8.59%
lat (msec) : 2=13.28%, 10=9.38%, 20=9.38%, 50=3.91%
cpu : usr=2.56%, sys=0.00%, ctx=26, majf=2, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=68.8%, 16=31.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.1%, 8=5.0%, 16=1.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2384: Mon Oct 11 16:33:05 2021
write: IOPS=3121, BW=24.4MiB/s (25.6MB/s)(1024KiB/41msec); 0 zone resets
clat (usec): min=3, max=14454, avg=233.47, stdev=1499.03
lat (usec): min=3, max=14454, avg=233.55, stdev=1499.06
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 5], 90.00th=[ 6], 95.00th=[ 122],
| 99.00th=[ 8455], 99.50th=[14484], 99.90th=[14484], 99.95th=[14484],
| 99.99th=[14484]
lat (usec) : 4=22.66%, 10=71.09%, 50=0.78%, 250=0.78%, 750=0.78%
lat (msec) : 2=1.56%, 4=0.78%, 10=0.78%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=13, majf=1, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2385: Mon Oct 11 16:33:05 2021
write: IOPS=3200, BW=25.0MiB/s (26.2MB/s)(1024KiB/40msec); 0 zone resets
clat (usec): min=3, max=11767, avg=214.75, stdev=1229.67
lat (usec): min=3, max=11767, avg=214.82, stdev=1229.69
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 130],
| 99.00th=[ 5014], 99.50th=[11731], 99.90th=[11731], 99.95th=[11731],
| 99.99th=[11731]
lat (usec) : 4=21.88%, 10=69.53%, 20=0.78%, 50=1.56%, 100=0.78%
lat (usec) : 250=0.78%, 750=0.78%
lat (msec) : 2=0.78%, 4=0.78%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=12, majf=2, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2386: Mon Oct 11 16:33:05 2021
write: IOPS=2976, BW=23.3MiB/s (24.4MB/s)(1024KiB/43msec); 0 zone resets
clat (usec): min=3, max=12613, avg=270.36, stdev=1358.23
lat (usec): min=3, max=12613, avg=270.44, stdev=1358.28
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 1188],
| 99.00th=[ 6063], 99.50th=[12649], 99.90th=[12649], 99.95th=[12649],
| 99.99th=[12649]
lat (usec) : 4=10.94%, 10=79.69%, 20=1.56%, 50=0.78%, 250=0.78%
lat (usec) : 750=0.78%
lat (msec) : 2=0.78%, 4=2.34%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=16, majf=5, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2387: Mon Oct 11 16:33:05 2021
write: IOPS=1600, BW=12.5MiB/s (13.1MB/s)(1024KiB/80msec); 0 zone resets
clat (usec): min=3, max=13691, avg=554.03, stdev=1567.65
lat (usec): min=3, max=13691, avg=554.20, stdev=1567.66
clat percentiles (usec):
| 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 10],
| 30.00th=[ 77], 40.00th=[ 99], 50.00th=[ 135], 60.00th=[ 174],
| 70.00th=[ 208], 80.00th=[ 392], 90.00th=[ 1057], 95.00th=[ 2802],
| 99.00th=[ 7111], 99.50th=[13698], 99.90th=[13698], 99.95th=[13698],
| 99.99th=[13698]
lat (usec) : 4=0.78%, 10=20.31%, 20=2.34%, 50=1.56%, 100=17.97%
lat (usec) : 250=30.47%, 500=9.38%, 750=3.91%, 1000=2.34%
lat (msec) : 2=3.91%, 4=3.91%, 10=2.34%, 20=0.78%
cpu : usr=1.27%, sys=2.53%, ctx=125, majf=4, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2388: Mon Oct 11 16:33:05 2021
write: IOPS=2976, BW=23.3MiB/s (24.4MB/s)(1024KiB/43msec); 0 zone resets
clat (usec): min=3, max=11465, avg=238.97, stdev=1232.98
lat (usec): min=3, max=11465, avg=239.06, stdev=1233.02
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 10], 95.00th=[ 619],
| 99.00th=[ 5342], 99.50th=[11469], 99.90th=[11469], 99.95th=[11469],
| 99.99th=[11469]
lat (usec) : 4=10.94%, 10=81.25%, 20=0.78%, 500=0.78%, 750=1.56%
lat (msec) : 2=0.78%, 4=1.56%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=18, majf=5, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2389: Mon Oct 11 16:33:05 2021
write: IOPS=29.3k, BW=28.6MiB/s (30.0MB/s)(1024KiB/35msec); 0 zone resets
clat (nsec): min=52, max=4005.3k, avg=21454.89, stdev=228411.55
lat (nsec): min=87, max=4005.4k, avg=21563.28, stdev=228409.29
clat percentiles (nsec):
| 1.00th=[ 59], 5.00th=[ 90], 10.00th=[ 112],
| 20.00th=[ 181], 30.00th=[ 217], 40.00th=[ 282],
| 50.00th=[ 1304], 60.00th=[ 1496], 70.00th=[ 3216],
| 80.00th=[ 3536], 90.00th=[ 5152], 95.00th=[ 11584],
| 99.00th=[ 272384], 99.50th=[1482752], 99.90th=[3620864],
| 99.95th=[4014080], 99.99th=[4014080]
lat (nsec) : 100=7.71%, 250=29.88%, 500=5.47%, 750=0.39%, 1000=0.20%
lat (usec) : 2=21.19%, 4=20.41%, 10=7.81%, 20=4.88%, 50=0.49%
lat (usec) : 100=0.20%, 250=0.29%, 500=0.39%, 750=0.10%
lat (msec) : 2=0.20%, 4=0.29%, 10=0.10%
cpu : usr=8.82%, sys=0.00%, ctx=28, majf=582, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2390: Mon Oct 11 16:33:05 2021
write: IOPS=29.3k, BW=28.6MiB/s (30.0MB/s)(1024KiB/35msec); 0 zone resets
clat (nsec): min=50, max=3672.6k, avg=21308.01, stdev=213632.75
lat (nsec): min=85, max=3672.7k, avg=21416.38, stdev=213630.86
clat percentiles (nsec):
| 1.00th=[ 56], 5.00th=[ 75], 10.00th=[ 104],
| 20.00th=[ 177], 30.00th=[ 211], 40.00th=[ 378],
| 50.00th=[ 1304], 60.00th=[ 2928], 70.00th=[ 3248],
| 80.00th=[ 3632], 90.00th=[ 5472], 95.00th=[ 11840],
| 99.00th=[ 296960], 99.50th=[1515520], 99.90th=[2932736],
| 99.95th=[3686400], 99.99th=[3686400]
lat (nsec) : 100=9.38%, 250=26.95%, 500=5.37%, 750=0.39%
lat (usec) : 2=16.70%, 4=25.68%, 10=8.89%, 20=4.69%, 50=0.88%
lat (usec) : 500=0.20%, 750=0.20%
lat (msec) : 2=0.20%, 4=0.49%
cpu : usr=8.82%, sys=0.00%, ctx=20, majf=599, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2391: Mon Oct 11 16:33:05 2021
write: IOPS=41.0k, BW=40.0MiB/s (41.9MB/s)(1024KiB/25msec); 0 zone resets
clat (nsec): min=49, max=2654.1k, avg=8722.09, stdev=117739.66
lat (nsec): min=84, max=2654.2k, avg=8769.59, stdev=117739.76
clat percentiles (nsec):
| 1.00th=[ 51], 5.00th=[ 58], 10.00th=[ 74],
| 20.00th=[ 96], 30.00th=[ 114], 40.00th=[ 147],
| 50.00th=[ 205], 60.00th=[ 1224], 70.00th=[ 1320],
| 80.00th=[ 1432], 90.00th=[ 1848], 95.00th=[ 2416],
| 99.00th=[ 10048], 99.50th=[ 411648], 99.90th=[1990656],
| 99.95th=[2637824], 99.99th=[2637824]
lat (nsec) : 50=0.20%, 100=22.07%, 250=28.81%, 500=2.25%, 750=0.10%
lat (usec) : 2=38.09%, 4=4.88%, 10=2.54%, 20=0.29%, 50=0.10%
lat (usec) : 500=0.29%, 750=0.10%
lat (msec) : 2=0.20%, 4=0.10%
cpu : usr=4.17%, sys=0.00%, ctx=13, majf=477, minf=21
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2392: Mon Oct 11 16:33:05 2021
write: IOPS=33.0k, BW=32.3MiB/s (33.8MB/s)(1024KiB/31msec); 0 zone resets
clat (nsec): min=53, max=3093.6k, avg=17587.87, stdev=179151.15
lat (nsec): min=91, max=3093.7k, avg=17708.65, stdev=179151.47
clat percentiles (nsec):
| 1.00th=[ 56], 5.00th=[ 59], 10.00th=[ 80],
| 20.00th=[ 181], 30.00th=[ 241], 40.00th=[ 282],
| 50.00th=[ 1240], 60.00th=[ 3216], 70.00th=[ 3504],
| 80.00th=[ 3792], 90.00th=[ 5472], 95.00th=[ 11072],
| 99.00th=[ 189440], 99.50th=[ 937984], 99.90th=[2736128],
| 99.95th=[3096576], 99.99th=[3096576]
lat (nsec) : 100=12.40%, 250=20.12%, 500=15.14%, 750=0.39%
lat (usec) : 2=9.67%, 4=24.22%, 10=10.84%, 20=5.47%, 50=0.68%
lat (usec) : 250=0.20%, 500=0.10%, 750=0.10%, 1000=0.20%
lat (msec) : 2=0.10%, 4=0.39%
cpu : usr=0.00%, sys=13.33%, ctx=18, majf=531, minf=21
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2393: Mon Oct 11 16:33:05 2021
write: IOPS=39.4k, BW=38.5MiB/s (40.3MB/s)(1024KiB/26msec); 0 zone resets
clat (nsec): min=50, max=3405.8k, avg=9250.34, stdev=122881.30
lat (nsec): min=85, max=3405.8k, avg=9331.42, stdev=122880.20
clat percentiles (nsec):
| 1.00th=[ 55], 5.00th=[ 59], 10.00th=[ 70],
| 20.00th=[ 85], 30.00th=[ 99], 40.00th=[ 117],
| 50.00th=[ 215], 60.00th=[ 1272], 70.00th=[ 1416],
| 80.00th=[ 1576], 90.00th=[ 4576], 95.00th=[ 5472],
| 99.00th=[ 34560], 99.50th=[ 399360], 99.90th=[1155072],
| 99.95th=[3391488], 99.99th=[3391488]
lat (nsec) : 100=30.66%, 250=20.41%, 500=4.69%, 750=0.49%
lat (usec) : 2=27.73%, 4=3.81%, 10=9.47%, 20=1.46%, 50=0.49%
lat (usec) : 250=0.10%, 500=0.20%, 750=0.10%, 1000=0.10%
lat (msec) : 2=0.20%, 4=0.10%
cpu : usr=8.00%, sys=0.00%, ctx=18, majf=450, minf=20
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2394: Mon Oct 11 16:33:05 2021
write: IOPS=7757, BW=30.1MiB/s (31.5MB/s)(1016KiB/33msec); 0 zone resets
clat (usec): min=23, max=3399, avg=49.62, stdev=216.57
lat (usec): min=23, max=3399, avg=49.70, stdev=216.57
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 25], 20.00th=[ 26],
| 30.00th=[ 27], 40.00th=[ 28], 50.00th=[ 30], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 49],
| 99.00th=[ 388], 99.50th=[ 578], 99.90th=[ 3392], 99.95th=[ 3392],
| 99.99th=[ 3392]
lat (usec) : 50=94.92%, 100=1.56%, 250=1.17%, 500=0.78%, 750=0.39%
lat (msec) : 4=0.39%
cpu : usr=0.00%, sys=25.00%, ctx=20, majf=256, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=2, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 0: pid=2395: Mon Oct 11 16:33:05 2021
write: IOPS=14.2k, BW=55.6MiB/s (58.3MB/s)(1024KiB/18msec); 0 zone resets
clat (usec): min=24, max=2564, avg=60.04, stdev=225.97
lat (usec): min=24, max=2564, avg=60.14, stdev=225.97
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28],
| 30.00th=[ 30], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 34],
| 70.00th=[ 35], 80.00th=[ 39], 90.00th=[ 44], 95.00th=[ 50],
| 99.00th=[ 742], 99.50th=[ 2409], 99.90th=[ 2573], 99.95th=[ 2573],
| 99.99th=[ 2573]
lat (usec) : 50=94.92%, 100=2.34%, 250=0.39%, 500=0.78%, 750=0.78%
lat (msec) : 4=0.78%
cpu : usr=0.00%, sys=47.06%, ctx=27, majf=261, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2396: Mon Oct 11 16:33:05 2021
write: IOPS=12.2k, BW=47.4MiB/s (49.7MB/s)(1020KiB/21msec); 0 zone resets
clat (usec): min=25, max=4368, avg=70.05, stdev=295.82
lat (usec): min=25, max=4368, avg=70.18, stdev=295.83
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 36], 60.00th=[ 38],
| 70.00th=[ 40], 80.00th=[ 44], 90.00th=[ 74], 95.00th=[ 88],
| 99.00th=[ 857], 99.50th=[ 1680], 99.90th=[ 4359], 99.95th=[ 4359],
| 99.99th=[ 4359]
lat (usec) : 50=82.81%, 100=14.45%, 250=0.78%, 750=0.39%, 1000=0.39%
lat (msec) : 2=0.39%, 10=0.39%
cpu : usr=15.00%, sys=35.00%, ctx=18, majf=264, minf=23
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=1, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2397: Mon Oct 11 16:33:05 2021
write: IOPS=7529, BW=29.1MiB/s (30.5MB/s)(1012KiB/34msec); 0 zone resets
clat (usec): min=23, max=3350, avg=66.34, stdev=234.18
lat (usec): min=23, max=3350, avg=66.48, stdev=234.18
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 38],
| 70.00th=[ 42], 80.00th=[ 47], 90.00th=[ 53], 95.00th=[ 78],
| 99.00th=[ 1029], 99.50th=[ 1172], 99.90th=[ 3359], 99.95th=[ 3359],
| 99.99th=[ 3359]
lat (usec) : 50=86.72%, 100=7.81%, 250=1.17%, 500=1.56%, 750=0.39%
lat (msec) : 2=0.78%, 4=0.39%
cpu : usr=0.00%, sys=30.30%, ctx=19, majf=256, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=3, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2398: Mon Oct 11 16:33:05 2021
write: IOPS=5953, BW=23.0MiB/s (24.1MB/s)(1012KiB/43msec); 0 zone resets
clat (usec): min=25, max=4743, avg=85.07, stdev=312.28
lat (usec): min=25, max=4743, avg=85.33, stdev=312.27
clat percentiles (usec):
| 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 35],
| 30.00th=[ 37], 40.00th=[ 44], 50.00th=[ 49], 60.00th=[ 55],
| 70.00th=[ 60], 80.00th=[ 65], 90.00th=[ 80], 95.00th=[ 141],
| 99.00th=[ 881], 99.50th=[ 1090], 99.90th=[ 4752], 99.95th=[ 4752],
| 99.99th=[ 4752]
lat (usec) : 50=50.78%, 100=42.19%, 250=2.73%, 500=1.17%, 750=0.78%
lat (usec) : 1000=0.39%
lat (msec) : 2=0.39%, 10=0.39%
cpu : usr=7.14%, sys=21.43%, ctx=34, majf=257, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=3, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=333MiB/s (350MB/s), 66.7MiB/s-90.9MiB/s (69.9MB/s-95.3MB/s), io=5120KiB (5243kB), run=11-15msec
WRITE: bw=375MiB/s (393MB/s), 12.5MiB/s-55.6MiB/s (13.1MB/s-58.3MB/s), io=30.0MiB (31.4MB), run=18-80msec
Disk stats (read/write):
zram1: ios=549/3114, merge=0/0, ticks=0/40, in_queue=40, util=29.07%
fio: file hash not empty on exit
QA output created by 095
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=45056, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=133120, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=51200, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=342016, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=275456, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=416768, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=233472, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=247808, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=177152, buflen=1024
fio: io_u_queue.h:37: io_u_qpush: Assertion '0' failed.
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=30720, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=92160, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=158720, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=513024, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=33792, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=449536, buflen=1024
fio: io_u_queue.h:37: io_u_qpush: Assertion '0' failed.
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=43008, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=114688, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=241664, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=115712, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=370688, buflen=1024
fio: io_u_queue.h:37: io_u_qpush: Assertion '0' failed.
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=222208, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=43008, buflen=1024
fio: pid=2364, got signal=6
fio: pid=2367, got signal=6
fio: pid=2368, got signal=6
fio: file hash not empty on exit
Silence is golden
