Hi Nicholas
Previous message was sent from the wrong address, sorry. Replies are inline.
> What Linux initiator are you using that supports MC/S btw..?
iscsi-initiator-utils-6.2.0.873-29.el7.x86_64 (open-iscsi.org) I have
replaced the default iscsi-initiator-utils in Fedora 20
with a package from Centos 7. Though it was a good idea to eliminate
it as a potential cause.
I have no experience with MC/S (multipathd has served me well) so
maybe this is PEBKAC?
Commands that I used:
[root@localhost ~]# iscsiadm -m discovery -t st -p 192.168.5.1 -I p32p1
192.168.5.1:3260,1 iqn.2015-09.com.total-adv.local.storage:iscsi
[root@localhost ~]# iscsiadm -m node -T
iqn.2015-09.com.total-adv.local.storage:iscsi -l
Logging in to [iface: p32p1, target:
iqn.2015-09.com.total-adv.local.storage:iscsi, portal:
192.168.5.1,3260] (multiple)
Login to [iface: p32p1, target:
iqn.2015-09.com.total-adv.local.storage:iscsi, portal:
192.168.5.1,3260] successful.
[root@localhost ~]# iscsiadm --mode session
tcp: [1] 192.168.5.1:3260,1
iqn.2015-09.com.total-adv.local.storage:iscsi (non-flash)
[root@localhost ~]# iscsiadm --mode session -r 1 --op new
Logging in to [iface: p32p1, target:
iqn.2015-09.com.total-adv.local.storage:iscsi, portal:
192.168.5.1,3260] (multiple)
Login to [iface: p32p1, target:
iqn.2015-09.com.total-adv.local.storage:iscsi, portal:
192.168.5.1,3260] successful.
[root@localhost ~]# iscsiadm --mode session
tcp: [1] 192.168.5.1:3260,1
iqn.2015-09.com.total-adv.local.storage:iscsi (non-flash)
tcp: [2] 192.168.5.1:3260,1
iqn.2015-09.com.total-adv.local.storage:iscsi (non-flash)
[root@localhost ~]# lsscsi | grep "LIO"
[10:0:0:0] disk LIO-ORG rd0 4.0 /dev/sdc
[11:0:0:0] disk LIO-ORG rd0 4.0 /dev/sdd
iscsiadm -m session -P 3 output is attached in iscsi_session.txt
> So looking at the performance results, I'm a bit confused..
Sorry I should have commented those results a bit better.
> [root@localhost ~]# lsscsi | grep "LIO"
> [11:0:0:0] disk LIO-ORG pv0 4.0 /dev/sdc
> [11:0:0:1] disk LIO-ORG lv0 4.0 /dev/sdf
> [11:0:0:2] disk LIO-ORG rd0 4.0 /dev/sde
> [11:0:0:3] disk LIO-ORG rd1 4.0 /dev/sdd
>
> ----------------------------- <IBLOCK + pv0>
>
> [root@localhost ~]# dd_rescue -ADv -m 16G -b 64k /dev/zero /dev/sdc
> dd_rescue: (info): ipos: 16773120.0k, opos: 16773120.0k, xferd: 16773120.0k
> errs: 0, errxfer: 0.0k, succxfer: 16773120.0k
> +curr.rate: 48464kB/s, avg.rate: 48443kB/s, avg.load: 3.3%
> >----------------------------------------.< 99% ETA: 0:00:00
> dd_rescue: (info): Summary for /dev/zero -> /dev/sdc
> dd_rescue: (info): ipos: 16777216.0k, opos: 16777216.0k, xferd: 16777216.0k
> errs: 0, errxfer: 0.0k, succxfer: 16777216.0k
> +curr.rate: 45179kB/s, avg.rate: 48439kB/s, avg.load: 3.3%
> >-----------------------------------------< 100% TOT: 0:05:46
>
> ----------------------------- <RAMDISK-MCP>
This is with NULLIO, /dev/sdd (rd1)
>
> [root@localhost ~]# dd_rescue -ADv -m 4G -b 64k /dev/zero /dev/sdd
> dd_rescue: (info): ipos: 4191232.0k, opos: 4191232.0k, xferd: 4191232.0k
> errs: 0, errxfer: 0.0k, succxfer: 4191232.0k
> +curr.rate: 62343kB/s, avg.rate: 62376kB/s, avg.load: 4.6%
> >----------------------------------------.< 99% ETA: 0:00:00
> dd_rescue: (info): Summary for /dev/zero -> /dev/sdd
> dd_rescue: (info): ipos: 4194304.0k, opos: 4194304.0k, xferd: 4194304.0k
> errs: 0, errxfer: 0.0k, succxfer: 4194304.0k
> +curr.rate: 61934kB/s, avg.rate: 62375kB/s, avg.load: 4.6%
> >-----------------------------------------< 100% TOT: 0:01:07
>
> ----------------------------- <RAMDISK-MCP + NULLIO>
This is ramdisk, /dev/sde (rd0)
The rest is correct, -D sets direct i/o.
>
> [root@localhost ~]# dd_rescue -ADv -m 4G -b 64k /dev/zero /dev/sde
> dd_rescue: (info): ipos: 4188160.0k, opos: 4188160.0k, xferd: 4188160.0k
> errs: 0, errxfer: 0.0k, succxfer: 4188160.0k
> +curr.rate: 62515kB/s, avg.rate: 61642kB/s, avg.load: 4.8%
> >----------------------------------------.< 99% ETA: 0:00:00
> dd_rescue: (info): Summary for /dev/zero -> /dev/sde
> dd_rescue: (info): ipos: 4194304.0k, opos: 4194304.0k, xferd: 4194304.0k
> errs: 0, errxfer: 0.0k, succxfer: 4194304.0k
> +curr.rate: 62403kB/s, avg.rate: 61643kB/s, avg.load: 4.8%
> >-----------------------------------------< 100% TOT: 0:01:08
>
> Buffered I/O
> ----------------------------- <Without O_DIRECT>
> [root@localhost ~]# dd_rescue -Av -m 16G -b 64k /dev/zero /dev/sdc
> dd_rescue: (info): ipos: 16776192.0k, opos: 16776192.0k, xferd: 16776192.0k
> errs: 0, errxfer: 0.0k, succxfer: 16776192.0k
> +curr.rate: 115223kB/s, avg.rate: 136634kB/s, avg.load: 14.7%
> >----------------------------------------.< 99% ETA: 0:00:00
> dd_rescue: (info): Summary for /dev/zero -> /dev/sdc
> dd_rescue: (info): ipos: 16777216.0k, opos: 16777216.0k, xferd: 16777216.0k
> errs: 0, errxfer: 0.0k, succxfer: 16777216.0k
> +curr.rate: 86427kB/s, avg.rate: 112677kB/s, avg.load: 12.5%
> >-----------------------------------------< 100% TOT: 0:02:29
>
> If RAMDISK-MCP + O_DIRECT is not able to reach ~110 MB/sec up to the
> limit of a unidirectional 1 Gb/sec port, it would indicate there is
> something else going on outside of iscsi application layer code.
I eliminated everything else that I thought of, we can walk try it all again.
> So I'd recommend first verifying the network using iperf that normal
> socket I/O is able to reach 1 Gb/sec in both directions.
This was done on initial server setup but I ran the tests again.
Output from iperf and iperf3 is attached in iperf_tests.txt
Network cards used: HP NC364T, Intel 82571EB, on client and server.
While the iperf was running I monitored the network throughput with dstat.
Bandwith was about 118MB/s even with bidirectional test so I guess the
network can be ruled out?
>
> From there, I'd recommend using fio with different iodepth settings +
> direct=1 in order to determine at which point you're able to saturate
> the 1 Gb/sec link with blocksize=64k.
>
> I'd recommend setting 'echo noop > /sys/block/$DEV/queue/scheduler'
> on the Linux iscsi initiator LUNs as well.
>
> Also keep in mind that for sequential I/O, the initiator side will be
> doing merging of 64k to larger requests. You can run 'iostat -xm 2'
> on both sides to see what I/O sizes are actually being generated.
Iostat on client and server report an avgrq-sz of 128 consistently so I'm not
seeing that. Kernel version on client is 4.1.5 if that makes any difference.
I have done the suggested tests, scheduler was set to noop.
I have used ramdisk and a partition on a hard drive as backstores.
Results from the test are summarized below, full test output is
attached in fio_scsi_tests.txt
iodepth 1, ramdisk, 64k block size, bandwith 62MB/s, client iostat avgrq-sz 128
iodepth 2, ramdisk, 64k block size, bandwith 112MB/s, client iostat avgrq-sz 128
iodepth 3, ramdisk, 64k block size, bandwith 120MB/s, client iostat avgrq-sz 128
I have also run tests with iodepth set at 1 and changed the block size
iodepth 1, ramdisk, 128k block size, bandwith 63MB/s, client iostat avgrq-sz 256
iodepth 1, ramdisk, 256k block size, bandwith 80MB/s, client iostat avgrq-sz 512
iodepth 1, ramdisk, 512k block size, bandwith 88MB/s, client iostat
avgrq-sz 1024
iodepth 1, ramdisk, 1024k block size, bandwith 91MB/s, client iostat
avgrq-sz 2048
iodepth 1, ramdisk, 4096k block size, bandwith 93MB/s, client iostat
avgrq-sz 8192
iodepth 1, ramdisk, 8192k block size, bandwith 114MB/s, client iostat
avgrq-sz 8192
iodepth 1, ramdisk, 16384k block size, bandwith 116MB/s, client iostat
avgrq-sz 8192
Hard drive as backstore:
iodepth 1, block, 64k block size, bandwith 50MB/s, client iostat
avgrq-sz 128, server iostat avgrq-sz 128
iodepth 2, block, 64k block size, bandwith 94MB/s, client iostat
avgrq-sz 128, server iostat avgrq-sz 128
iodepth 3, block, 64k block size, bandwith 113MB/s, client iostat
avgrq-sz 128, server iostat avgrq-sz 128
iodepth 4, block, 64k block size, bandwith 118MB/s, client iostat
avgrq-sz 128, server iostat avgrq-sz 128
Changing the block size has some effect but unfortunately bandwidth
maxes at 70MB/s even when block size
was set at 32768K. At this setting avgrq-sz on server was 1024 and
on the client 8192
It's 4:30am here so I didn't have the time to compile a better report.
> Sep 17 02:08:08 storage kernel: Got unknown iSCSI OpCode: 0x43
> Sep 17 02:08:08 storage kernel: Cannot recover from unknown opcode while ERL=0, closing iSCSI connection.
>
>
> Taking a look at this now.
Thanks
> So this one is was fixed for mainline in 05-2014, and picked up by linux-3.10.y stable here:
>
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/drivers/target/iscsi?h=linux-3.10.y&id=d5c55fa31a2947ac517abddd7ca6975a17bef178
>
> but it looks like it was not picked up for RHEL 7.1 / 3.10.0-229.14.1.el7.x86_64 code.
I should probably post an error report at redhat bugzilla.
Thank you for help
Dragan
[root@localhost ~]# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 6.2.0.873-28
Target: iqn.2015-09.com.total-adv.local.storage:iscsi (non-flash)
Current Portal: 192.168.5.1:3260,1
Persistent Portal: 192.168.5.1:3260,1
**********
Interface:
**********
Iface Name: p32p1
Iface Transport: tcp
Iface Initiatorname: iqn.2015-09.com.total-adv.local.storage:staletic
Iface IPaddress: 192.168.5.2
Iface HWaddress: 00:24:81:7b:e7:6d
Iface Netdev: p32p1
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 120
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 262144
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 10 State: running
scsi10 Channel 00 Id 0 Lun: 0
Attached scsi disk sdc State: running
**********
Interface:
**********
Iface Name: p32p1
Iface Transport: tcp
Iface Initiatorname: iqn.2015-09.com.total-adv.local.storage:staletic
Iface IPaddress: 192.168.5.2
Iface HWaddress: 00:24:81:7b:e7:6d
Iface Netdev: p32p1
SID: 2
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 120
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 262144
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 11 State: running
scsi11 Channel 00 Id 0 Lun: 0
Attached scsi disk sdd State: running
[root@localhost ~]# iperf3 -Vc 192.168.5.1
iperf 3.0.11
Linux localhost.localdomain 4.1.5-100.fc21.x86_64 #1 SMP Tue Aug 11 00:24:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Time: Thu, 17 Sep 2015 23:29:17 GMT
Connecting to host 192.168.5.1, port 5201
Cookie: localhost.localdomain.1442532557.671
TCP MSS: 8948 (default)
[ 4] local 192.168.5.2 port 52135 connected to 192.168.5.1 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 119 MBytes 996 Mbits/sec 0 315 KBytes
[ 4] 1.00-2.00 sec 118 MBytes 989 Mbits/sec 0 315 KBytes
[ 4] 2.00-3.00 sec 118 MBytes 989 Mbits/sec 0 315 KBytes
[ 4] 3.00-4.00 sec 118 MBytes 990 Mbits/sec 0 315 KBytes
[ 4] 4.00-5.00 sec 118 MBytes 991 Mbits/sec 0 315 KBytes
[ 4] 5.00-6.00 sec 119 MBytes 995 Mbits/sec 0 472 KBytes
[ 4] 6.00-7.00 sec 118 MBytes 987 Mbits/sec 0 472 KBytes
[ 4] 7.00-8.00 sec 118 MBytes 991 Mbits/sec 0 472 KBytes
[ 4] 8.00-9.00 sec 118 MBytes 990 Mbits/sec 0 472 KBytes
[ 4] 9.00-10.00 sec 118 MBytes 989 Mbits/sec 0 472 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.15 GBytes 991 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.15 GBytes 990 Mbits/sec receiver
CPU Utilization: local/sender 5.6% (0.3%u/5.3%s), remote/receiver 0.8% (0.1%u/0.7%s)
[root@localhost ~]# iperf3 -VRc 192.168.5.1
iperf 3.0.11
Linux localhost.localdomain 4.1.5-100.fc21.x86_64 #1 SMP Tue Aug 11 00:24:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Time: Thu, 17 Sep 2015 23:30:27 GMT
Connecting to host 192.168.5.1, port 5201
Reverse mode, remote host 192.168.5.1 is sending
Cookie: localhost.localdomain.1442532627.502
TCP MSS: 8948 (default)
[ 4] local 192.168.5.2 port 52145 connected to 192.168.5.1 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 118 MBytes 991 Mbits/sec
[ 4] 1.00-2.00 sec 118 MBytes 990 Mbits/sec
[ 4] 2.00-3.00 sec 118 MBytes 990 Mbits/sec
[ 4] 3.00-4.00 sec 118 MBytes 990 Mbits/sec
[ 4] 4.00-5.00 sec 118 MBytes 990 Mbits/sec
[ 4] 5.00-6.00 sec 118 MBytes 990 Mbits/sec
[ 4] 6.00-7.00 sec 118 MBytes 990 Mbits/sec
[ 4] 7.00-8.00 sec 118 MBytes 990 Mbits/sec
[ 4] 8.00-9.00 sec 118 MBytes 990 Mbits/sec
[ 4] 9.00-10.00 sec 118 MBytes 990 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.15 GBytes 991 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.15 GBytes 990 Mbits/sec receiver
CPU Utilization: local/receiver 16.8% (1.3%u/15.5%s), remote/sender 0.4% (0.0%u/0.4%s)
[root@localhost ~]# iperf -c 192.168.5.1 -m -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.5.1, TCP port 5001
TCP window size: 650 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.5.2 port 48087 connected with 192.168.5.1 port 5001
[ 4] local 192.168.5.2 port 5001 connected with 192.168.5.1 port 44549
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.15 GBytes 987 Mbits/sec
[ 5] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
[ 4] 0.0-10.0 sec 1.15 GBytes 986 Mbits/sec
[ 4] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
iodepth 1, ramdisk, 64k block size, bandwith 62MB/s, client iostat avgrq-sz 128
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=64k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/62848KB/0KB /s] [0/982/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=8494: Fri Sep 18 03:01:31 2015
write: io=4096.0MB, bw=62130KB/s, iops=970, runt= 67508msec
slat (usec): min=8, max=237, avg=33.68, stdev= 9.63
clat (usec): min=711, max=1805, avg=993.55, stdev=83.39
lat (usec): min=736, max=1839, avg=1027.63, stdev=84.61
clat percentiles (usec):
| 1.00th=[ 908], 5.00th=[ 940], 10.00th=[ 948], 20.00th=[ 956],
| 30.00th=[ 964], 40.00th=[ 964], 50.00th=[ 964], 60.00th=[ 972],
| 70.00th=[ 972], 80.00th=[ 980], 90.00th=[ 1192], 95.00th=[ 1208],
| 99.00th=[ 1224], 99.50th=[ 1240], 99.90th=[ 1256], 99.95th=[ 1288],
| 99.99th=[ 1432]
bw (KB /s): min=60544, max=63616, per=100.00%, avg=62191.93, stdev=490.59
lat (usec) : 750=0.25%, 1000=86.05%
lat (msec) : 2=13.70%
cpu : usr=1.06%, sys=3.79%, ctx=66075, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=62130KB/s, minb=62130KB/s, maxb=62130KB/s, mint=67508msec, maxt=67508msec
Disk stats (read/write):
sdc: ios=59/65488, merge=0/0, ticks=26/65035, in_queue=65024, util=96.33%
iodepth 2, ramdisk, 64k block size, bandwith 112MB/s, client iostat avgrq-sz 128
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=64k --direct=1 --rw=write --iodepth=2 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/110.8MB/0KB /s] [0/1761/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=8539: Fri Sep 18 03:03:20 2015
write: io=4096.0MB, bw=112981KB/s, iops=1765, runt= 37124msec
slat (usec): min=7, max=1312, avg=29.97, stdev=11.59
clat (usec): min=720, max=1993, avg=1100.28, stdev=144.68
lat (usec): min=735, max=2313, avg=1130.64, stdev=144.31
clat percentiles (usec):
| 1.00th=[ 924], 5.00th=[ 948], 10.00th=[ 956], 20.00th=[ 964],
| 30.00th=[ 972], 40.00th=[ 980], 50.00th=[ 1064], 60.00th=[ 1208],
| 70.00th=[ 1224], 80.00th=[ 1224], 90.00th=[ 1240], 95.00th=[ 1256],
| 99.00th=[ 1480], 99.50th=[ 1496], 99.90th=[ 1512], 99.95th=[ 1528],
| 99.99th=[ 1736]
bw (KB /s): min=109696, max=116224, per=100.00%, avg=113056.86, stdev=1488.06
lat (usec) : 750=0.04%, 1000=48.73%
lat (msec) : 2=51.22%
cpu : usr=1.99%, sys=6.02%, ctx=66324, majf=0, minf=7
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=112980KB/s, minb=112980KB/s, maxb=112980KB/s, mint=37124msec, maxt=37124msec
Disk stats (read/write):
sdc: ios=60/65201, merge=0/0, ticks=29/70438, in_queue=70414, util=99.89%
iodepth 3, ramdisk, 64k block size, bandwith 120MB/s, client iostat avgrq-sz 128
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=64k --direct=1 --rw=write --iodepth=3 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=3
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/117.5MB/0KB /s] [0/1879/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=8546: Fri Sep 18 03:04:24 2015
write: io=4096.0MB, bw=120256KB/s, iops=1879, runt= 34878msec
slat (usec): min=6, max=242, avg=30.31, stdev=10.26
clat (usec): min=865, max=2418, avg=1563.59, stdev=165.78
lat (usec): min=906, max=2443, avg=1594.30, stdev=164.59
clat percentiles (usec):
| 1.00th=[ 1192], 5.00th=[ 1336], 10.00th=[ 1384], 20.00th=[ 1416],
| 30.00th=[ 1448], 40.00th=[ 1464], 50.00th=[ 1544], 60.00th=[ 1640],
| 70.00th=[ 1672], 80.00th=[ 1720], 90.00th=[ 1752], 95.00th=[ 1848],
| 99.00th=[ 1976], 99.50th=[ 1992], 99.90th=[ 2008], 99.95th=[ 2024],
| 99.99th=[ 2128]
bw (KB /s): min=120192, max=120448, per=100.00%, avg=120323.71, stdev=53.64
lat (usec) : 1000=0.01%
lat (msec) : 2=99.83%, 4=0.16%
cpu : usr=2.26%, sys=6.36%, ctx=66440, majf=0, minf=7
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=3
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=120256KB/s, minb=120256KB/s, maxb=120256KB/s, mint=34878msec, maxt=34878msec
Disk stats (read/write):
sdc: ios=60/65161, merge=0/0, ticks=26/100578, in_queue=100546, util=99.87%
iodepth 1, ramdisk, 128k block size, bandwith 63MB/s, client iostat avgrq-sz 256
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=128k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/62464KB/0KB /s] [0/488/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=8689: Fri Sep 18 03:19:50 2015
write: io=4096.0MB, bw=63129KB/s, iops=493, runt= 66440msec
slat (usec): min=12, max=256, avg=58.22, stdev=17.77
clat (usec): min=1576, max=4950, avg=1964.32, stdev=136.79
lat (usec): min=1655, max=5002, avg=2023.11, stdev=138.76
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1704], 10.00th=[ 1864], 20.00th=[ 1912],
| 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960],
| 70.00th=[ 1976], 80.00th=[ 2008], 90.00th=[ 2160], 95.00th=[ 2192],
| 99.00th=[ 2352], 99.50th=[ 2416], 99.90th=[ 2480], 99.95th=[ 2480],
| 99.99th=[ 3664]
bw (KB /s): min=61828, max=64512, per=100.00%, avg=63192.91, stdev=467.05
lat (msec) : 2=79.16%, 4=20.84%, 10=0.01%
cpu : usr=1.20%, sys=3.37%, ctx=32825, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=32768/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=63129KB/s, minb=63129KB/s, maxb=63129KB/s, mint=66440msec, maxt=66440msec
Disk stats (read/write):
sdc: ios=59/32652, merge=0/0, ticks=24/63785, in_queue=63757, util=96.24%
iodepth 1, ramdisk, 256k block size, bandwith 80MB/s, client iostat avgrq-sz 512
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=256k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=0): [W] [100.0% done] [0KB/79872KB/0KB /s] [0/312/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9103: Fri Sep 18 03:22:46 2015
write: io=4096.0MB, bw=80772KB/s, iops=315, runt= 51928msec
slat (usec): min=19, max=301, avg=80.22, stdev=25.82
clat (usec): min=2702, max=5544, avg=3084.05, stdev=152.25
lat (usec): min=2734, max=5643, avg=3164.84, stdev=154.11
clat percentiles (usec):
| 1.00th=[ 2832], 5.00th=[ 2864], 10.00th=[ 2896], 20.00th=[ 2928],
| 30.00th=[ 2960], 40.00th=[ 3024], 50.00th=[ 3152], 60.00th=[ 3152],
| 70.00th=[ 3184], 80.00th=[ 3184], 90.00th=[ 3216], 95.00th=[ 3376],
| 99.00th=[ 3440], 99.50th=[ 3440], 99.90th=[ 3504], 99.95th=[ 3568],
| 99.99th=[ 5408]
bw (KB /s): min=79712, max=82267, per=100.00%, avg=80847.74, stdev=496.46
lat (msec) : 4=99.99%, 10=0.01%
cpu : usr=1.07%, sys=2.49%, ctx=16417, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=16384/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=80771KB/s, minb=80771KB/s, maxb=80771KB/s, mint=51928msec, maxt=51928msec
Disk stats (read/write):
sdc: ios=59/16309, merge=0/0, ticks=26/50375, in_queue=50370, util=97.34%
iodepth 1, ramdisk, 512k block size, bandwith 88MB/s, client iostat avgrq-sz 1024
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=512k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/88064KB/0KB /s] [0/172/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9130: Fri Sep 18 03:24:40 2015
write: io=4096.0MB, bw=88012KB/s, iops=171, runt= 47656msec
slat (usec): min=30, max=372, avg=132.50, stdev=37.67
clat (usec): min=5121, max=7319, avg=5679.28, stdev=213.46
lat (usec): min=5250, max=7490, avg=5812.37, stdev=218.76
clat percentiles (usec):
| 1.00th=[ 5344], 5.00th=[ 5344], 10.00th=[ 5408], 20.00th=[ 5472],
| 30.00th=[ 5600], 40.00th=[ 5600], 50.00th=[ 5664], 60.00th=[ 5728],
| 70.00th=[ 5856], 80.00th=[ 5856], 90.00th=[ 5920], 95.00th=[ 6048],
| 99.00th=[ 6176], 99.50th=[ 6176], 99.90th=[ 6304], 99.95th=[ 6368],
| 99.99th=[ 7328]
bw (KB /s): min=86693, max=89396, per=100.00%, avg=88088.81, stdev=597.02
lat (msec) : 10=100.00%
cpu : usr=1.07%, sys=1.72%, ctx=8217, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=8192/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=88012KB/s, minb=88012KB/s, maxb=88012KB/s, mint=47656msec, maxt=47656msec
Disk stats (read/write):
sdc: ios=59/8153, merge=0/0, ticks=29/46222, in_queue=46224, util=97.30%
iodepth 1, ramdisk, 1024k block size, bandwith 91MB/s, client iostat avgrq-sz 2048
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=1024k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/92160KB/0KB /s] [0/90/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9172: Fri Sep 18 03:26:47 2015
write: io=4096.0MB, bw=91194KB/s, iops=89, runt= 45993msec
slat (usec): min=56, max=697, avg=226.22, stdev=68.47
clat (usec): min=10257, max=11949, avg=10996.24, stdev=248.05
lat (usec): min=10485, max=12044, avg=11223.12, stdev=261.97
clat percentiles (usec):
| 1.00th=[10560], 5.00th=[10688], 10.00th=[10688], 20.00th=[10688],
| 30.00th=[10816], 40.00th=[10944], 50.00th=[10944], 60.00th=[11072],
| 70.00th=[11200], 80.00th=[11200], 90.00th=[11328], 95.00th=[11456],
| 99.00th=[11584], 99.50th=[11712], 99.90th=[11712], 99.95th=[11840],
| 99.99th=[11968]
bw (KB /s): min=90352, max=92180, per=100.00%, avg=91281.57, stdev=480.03
lat (msec) : 20=100.00%
cpu : usr=0.82%, sys=1.43%, ctx=4142, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=4096/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=91194KB/s, minb=91194KB/s, maxb=91194KB/s, mint=45993msec, maxt=45993msec
Disk stats (read/write):
sdc: ios=59/4091, merge=0/0, ticks=29/44957, in_queue=44958, util=97.74%
iodepth 1, ramdisk, 4096k block size, bandwith 93MB/s, client iostat avgrq-sz 8192
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=4096k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/90021KB/0KB /s] [0/21/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9221: Fri Sep 18 03:31:15 2015
write: io=4096.0MB, bw=93602KB/s, iops=22, runt= 44810msec
slat (usec): min=237, max=1513, avg=827.80, stdev=213.66
clat (usec): min=41356, max=44560, avg=42922.66, stdev=587.18
lat (usec): min=41710, max=45479, avg=43751.35, stdev=661.48
clat percentiles (usec):
| 1.00th=[41728], 5.00th=[41728], 10.00th=[42240], 20.00th=[42240],
| 30.00th=[42752], 40.00th=[42752], 50.00th=[42752], 60.00th=[43264],
| 70.00th=[43264], 80.00th=[43264], 90.00th=[43776], 95.00th=[43776],
| 99.00th=[44288], 99.50th=[44288], 99.90th=[44288], 99.95th=[44800],
| 99.99th=[44800]
bw (KB /s): min=92564, max=95255, per=100.00%, avg=93702.51, stdev=510.24
lat (msec) : 50=100.00%
cpu : usr=0.90%, sys=1.07%, ctx=1034, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=1024/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=93601KB/s, minb=93601KB/s, maxb=93601KB/s, mint=44810msec, maxt=44810msec
Disk stats (read/write):
sdc: ios=59/1021, merge=0/3066, ticks=27/44130, in_queue=44152, util=98.67%
iodepth 1, ramdisk, 8192k block size, bandwith 114MB/s, client iostat avgrq-sz 8192
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=8192k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=8M-8M/8M-8M/8M-8M, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/111.1MB/0KB /s] [0/13/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9258: Fri Sep 18 03:33:40 2015
write: io=4096.0MB, bw=114398KB/s, iops=13, runt= 36664msec
slat (usec): min=551, max=3007, avg=1786.69, stdev=466.22
clat (usec): min=69333, max=70160, avg=69812.16, stdev=113.15
lat (usec): min=70274, max=72800, avg=71599.84, stdev=468.27
clat percentiles (usec):
| 1.00th=[69120], 5.00th=[69120], 10.00th=[70144], 20.00th=[70144],
| 30.00th=[70144], 40.00th=[70144], 50.00th=[70144], 60.00th=[70144],
| 70.00th=[70144], 80.00th=[70144], 90.00th=[70144], 95.00th=[70144],
| 99.00th=[70144], 99.50th=[70144], 99.90th=[70144], 99.95th=[70144],
| 99.99th=[70144]
bw (KB /s): min=113777, max=115380, per=100.00%, avg=114503.25, stdev=322.69
lat (msec) : 100=100.00%
cpu : usr=1.11%, sys=1.45%, ctx=537, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=512/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=114398KB/s, minb=114398KB/s, maxb=114398KB/s, mint=36664msec, maxt=36664msec
Disk stats (read/write):
sdc: ios=59/1016, merge=0/3054, ticks=25/70670, in_queue=70762, util=98.42%
iodepth 1, ramdisk, 16384k block size, bandwith 116MB/s, client iostat avgrq-sz 8192
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=16384k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=16M-16M/16M-16M/16M-16M, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/112.0MB/0KB /s] [0/7/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9271: Fri Sep 18 03:35:43 2015
write: io=4096.0MB, bw=116496KB/s, iops=7, runt= 36004msec
slat (usec): min=1069, max=4809, avg=3437.15, stdev=879.40
clat (usec): min=136918, max=137554, avg=137188.84, stdev=137.46
lat (usec): min=138100, max=141877, avg=140626.94, stdev=859.74
clat percentiles (msec):
| 1.00th=[ 137], 5.00th=[ 137], 10.00th=[ 137], 20.00th=[ 137],
| 30.00th=[ 137], 40.00th=[ 137], 50.00th=[ 137], 60.00th=[ 137],
| 70.00th=[ 139], 80.00th=[ 139], 90.00th=[ 139], 95.00th=[ 139],
| 99.00th=[ 139], 99.50th=[ 139], 99.90th=[ 139], 99.95th=[ 139],
| 99.99th=[ 139]
bw (KB /s): min=115992, max=117658, per=100.00%, avg=116596.53, stdev=387.11
lat (msec) : 250=100.00%
cpu : usr=1.12%, sys=1.36%, ctx=278, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=256/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=116495KB/s, minb=116495KB/s, maxb=116495KB/s, mint=36004msec, maxt=36004msec
Disk stats (read/write):
sdc: ios=60/1020, merge=0/3072, ticks=29/138686, in_queue=138895, util=98.54%
iodepth 1, block, 64k block size, bandwith 50MB/s, client iostat avgrq-sz 128, server iostat avgrq-sz 128
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=64k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/49920KB/0KB /s] [0/780/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9427: Fri Sep 18 03:51:07 2015
write: io=4096.0MB, bw=50016KB/s, iops=781, runt= 83860msec
slat (usec): min=8, max=1043, avg=34.16, stdev=12.08
clat (usec): min=185, max=8811, avg=1242.39, stdev=224.46
lat (usec): min=964, max=8851, avg=1276.96, stdev=224.83
clat percentiles (usec):
| 1.00th=[ 964], 5.00th=[ 1004], 10.00th=[ 1176], 20.00th=[ 1192],
| 30.00th=[ 1208], 40.00th=[ 1208], 50.00th=[ 1208], 60.00th=[ 1208],
| 70.00th=[ 1224], 80.00th=[ 1224], 90.00th=[ 1240], 95.00th=[ 1464],
| 99.00th=[ 2512], 99.50th=[ 2736], 99.90th=[ 2992], 99.95th=[ 3056],
| 99.99th=[ 3280]
bw (KB /s): min=47872, max=51072, per=100.00%, avg=50061.74, stdev=341.35
lat (usec) : 250=0.01%, 1000=4.90%
lat (msec) : 2=92.79%, 4=2.30%, 10=0.01%
cpu : usr=0.90%, sys=3.08%, ctx=66801, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=50015KB/s, minb=50015KB/s, maxb=50015KB/s, mint=83860msec, maxt=83860msec
Disk stats (read/write):
sdc: ios=60/65428, merge=0/0, ticks=34/80961, in_queue=80914, util=96.61%
iodepth 2, block, 64k block size, bandwith 94MB/s, client iostat avgrq-sz 128, server iostat avgrq-sz 128
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=64k --direct=1 --rw=write --iodepth=2 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=2
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/94656KB/0KB /s] [0/1479/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9454: Fri Sep 18 03:54:34 2015
write: io=4096.0MB, bw=94492KB/s, iops=1476, runt= 44388msec
slat (usec): min=6, max=1302, avg=30.95, stdev=11.60
clat (usec): min=930, max=9257, avg=1320.87, stdev=305.29
lat (usec): min=964, max=9299, avg=1352.22, stdev=305.23
clat percentiles (usec):
| 1.00th=[ 980], 5.00th=[ 1176], 10.00th=[ 1192], 20.00th=[ 1208],
| 30.00th=[ 1208], 40.00th=[ 1224], 50.00th=[ 1224], 60.00th=[ 1224],
| 70.00th=[ 1240], 80.00th=[ 1288], 90.00th=[ 1496], 95.00th=[ 1976],
| 99.00th=[ 2736], 99.50th=[ 2960], 99.90th=[ 3088], 99.95th=[ 3248],
| 99.99th=[ 3472]
bw (KB /s): min=91904, max=95744, per=100.00%, avg=94549.20, stdev=629.62
lat (usec) : 1000=2.36%
lat (msec) : 2=92.78%, 4=4.86%, 10=0.01%
cpu : usr=1.71%, sys=5.21%, ctx=65000, majf=0, minf=7
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=94491KB/s, minb=94491KB/s, maxb=94491KB/s, mint=44388msec, maxt=44388msec
Disk stats (read/write):
sdc: ios=60/65239, merge=0/0, ticks=31/85029, in_queue=84990, util=99.68%
iodepth 3, block, 64k block size, bandwith 113MB/s, client iostat avgrq-sz 128, server iostat avgrq-sz 128
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=64k --direct=1 --rw=write --iodepth=3 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=3
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/111.5MB/0KB /s] [0/1783/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9473: Fri Sep 18 03:56:51 2015
write: io=4096.0MB, bw=113815KB/s, iops=1778, runt= 36852msec
slat (usec): min=6, max=335, avg=31.08, stdev=11.40
clat (usec): min=957, max=19966, avg=1653.18, stdev=361.26
lat (usec): min=988, max=19996, avg=1684.64, stdev=360.50
clat percentiles (usec):
| 1.00th=[ 1192], 5.00th=[ 1240], 10.00th=[ 1336], 20.00th=[ 1432],
| 30.00th=[ 1464], 40.00th=[ 1496], 50.00th=[ 1576], 60.00th=[ 1656],
| 70.00th=[ 1704], 80.00th=[ 1816], 90.00th=[ 1976], 95.00th=[ 2416],
| 99.00th=[ 2928], 99.50th=[ 3184], 99.90th=[ 3632], 99.95th=[ 3696],
| 99.99th=[ 5152]
bw (KB /s): min=108288, max=114688, per=100.00%, avg=113893.08, stdev=715.50
lat (usec) : 1000=0.13%
lat (msec) : 2=90.60%, 4=9.26%, 10=0.01%, 20=0.01%
cpu : usr=1.98%, sys=6.38%, ctx=65646, majf=0, minf=7
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=3
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=113814KB/s, minb=113814KB/s, maxb=113814KB/s, mint=36852msec, maxt=36852msec
Disk stats (read/write):
sdc: ios=60/65226, merge=0/0, ticks=31/106821, in_queue=106791, util=99.87%
iodepth 4, block, 64k block size, bandwith 118MB/s, client iostat avgrq-sz 128, server iostat avgrq-sz 128
------------------------------------------------------------------------------------------------------------------------------------
[root@localhost ~]# fio --name=write --ioengine=libaio --bs=64k --direct=1 --rw=write --iodepth=4 --size=4G --filename=/dev/sdc
write: (g=0): rw=write, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=4
fio-2.1.10
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/115.5MB/0KB /s] [0/1847/0 iops] [eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=9482: Fri Sep 18 03:58:08 2015
write: io=4096.0MB, bw=118013KB/s, iops=1843, runt= 35541msec
slat (usec): min=6, max=251, avg=30.74, stdev=10.25
clat (usec): min=927, max=24761, avg=2135.72, stdev=418.46
lat (usec): min=938, max=24790, avg=2166.87, stdev=417.81
clat percentiles (usec):
| 1.00th=[ 1208], 5.00th=[ 1528], 10.00th=[ 1848], 20.00th=[ 1928],
| 30.00th=[ 1960], 40.00th=[ 1992], 50.00th=[ 2128], 60.00th=[ 2160],
| 70.00th=[ 2192], 80.00th=[ 2256], 90.00th=[ 2480], 95.00th=[ 2928],
| 99.00th=[ 3440], 99.50th=[ 3600], 99.90th=[ 3856], 99.95th=[ 3952],
| 99.99th=[13632]
bw (KB /s): min=109696, max=118784, per=100.00%, avg=118092.80, stdev=1044.65
lat (usec) : 1000=0.11%
lat (msec) : 2=41.59%, 4=58.27%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=2.16%, sys=6.46%, ctx=64856, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=118013KB/s, minb=118013KB/s, maxb=118013KB/s, mint=35541msec, maxt=35541msec
Disk stats (read/write):
sdc: ios=60/65330, merge=0/0, ticks=32/138279, in_queue=138244, util=99.89%
