get err 5

[آرش خاوری راد <arash.khavary@xxxxxxxxx>]

Hi i got this error when test FIO sequential write every time, in sequential read, random read, random write not got this.

my Linux distro is CentOS Linux release 7.5.1804 (Core)

and my kernel version is 3.10.0-327.10.1.el7.x86_64

My output is:

job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
fio-3.1
Starting 16 processes

job1: (groupid=0, jobs=16): err= 5 (file:io_u.c:1756, func=io_u error, error=Input/output error): pid=10664: Fri Sep  4 00:03:23 2015
  write: IOPS=319, BW=1279MiB/s (1341MB/s)(2559GiB/2048690msec)
    slat (usec): min=227, max=101577k, avg=16698.60, stdev=395259.01
    clat (msec): min=71, max=102013, avg=302.24, stdev=1524.49
     lat (msec): min=72, max=102014, avg=318.94, stdev=1574.94
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  161], 10.00th=[  180], 20.00th=[  209],
     | 30.00th=[  230], 40.00th=[  247], 50.00th=[  264], 60.00th=[  284],
     | 70.00th=[  305], 80.00th=[  330], 90.00th=[  376], 95.00th=[  430],
     | 99.00th=[  550], 99.50th=[  592], 99.90th=[  726], 99.95th=[  810],
     | 99.99th=[17113]
   bw (  KiB/s): min= 8308, max=403830, per=13.68%, avg=179223.36, stdev=54500.84, samples=15737
   iops        : min=    2, max=   98, avg=43.32, stdev=13.35, samples=15737
  lat (msec)   : 100=0.01%, 250=27.42%, 500=36.58%, 750=1.31%, 1000=0.03%
  lat (msec)   : 2000=0.01%, >=2000=0.02%
  cpu          : usr=16.84%, sys=4.74%, ctx=199315, majf=0, minf=117471
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=1221.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,655023,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1279MiB/s (1341MB/s), 1279MiB/s-1279MiB/s (1341MB/s-1341MB/s), io=2559GiB (2748GB), run=2048690-2048690msec

Disk stats (read/write):
  sdb: ios=87/12938655, merge=0/0, ticks=62/1728507436, in_queue=1728692357, util=100.00%
  sdc: ios=43/12688086, merge=0/0, ticks=12/1680848678, in_queue=1680921305, util=100.00%
  sdd: ios=43/12793381, merge=0/0, ticks=9/1726297058, in_queue=1726441852, util=100.00%
  sde: ios=34/12824943, merge=0/0, ticks=9/1726084912, in_queue=1726194592, util=100.00%
  sdf: ios=0/1816141, merge=0/0, ticks=0/288898653, in_queue=300747577, util=100.00%
  sdg: ios=0/10970577, merge=0/0, ticks=0/1759675137, in_queue=1759784380, util=100.00%

My parameter is :


[global]
readwrite=write
blocksize=4m
direct=1
#numjobs=1
iodepth=16
ioengine=libaio
#filename=/dev/sdx
refill_buffers
time_based
runtime=600
ramp_time=10800
group_reporting

[job1]
numjobs=3
filename=/dev/sdb


[job2]
numjobs=2
filename=/dev/sdc


[job3]
numjobs=3
filename=/dev/sdd


[job4]
numjobs=3
filename=/dev/sde


[job5]
numjobs=2
filename=/dev/sdf


[job6]
numjobs=3
filename=/dev/sdg


.

Why get err 5?

thanks