Hi all,
I don't know exactly which mailinglists to use for this one so I hope
i used the right ones.
I did some performance testing on a new system and found out some
things I couldn't explain or didn't expect.
At the end are some questions I hope to get answered to explain the
tings i'm seeing in the test.
First of all a description of my setup.
The server is a 36 bay 3,5" supermicro chassis filled with 36x 2TB
SATA 7200 RPM disks. I use a single SSD as OS disk (Debian Squeeze
stable) connected to the motherboard SATA port1. The data disks are
connected to a LSI 9211-8i controller in IT mode (non-raid).
Once booted I created 3 raid6 MD devices of 10 disks each (16TB netto
each) with 6 global hotspares in the same sparegroup.
All MD devices have a chunk size of 64KB
I started testing using fio (2.0.4)
I used a bandwidth random read test I found on the Fusion IO website.
after every test i ran: sync; echo 3 >/proc/sys/vm/drop_caches;
The first test was on a single md device.
fio --name=test1 --ioengine=sync --direct=1 --rw=randread --bs=1m
--runtime=10 --filename=/dev/md0 --iodepth=1 --invalidate=1
(check the attachments for the full fio output, i'll only post the (in
my eyes) important snippets).
read : io=518144KB, bw=51726KB/s, iops=50 , runt= 10017msec
Disk stats (read/write):
md0: ios=8000/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=674/0, aggrmerge=0/0, aggrticks=10183/0, aggrin_queue=10183,
aggrutil=77.97%
sdc: ios=830/0, merge=0/0, ticks=12584/0, in_queue=12584, util=63.15%
So the IOPS for this test is 50. Bandwidth is around 50MB/s (50 iops x
1m blocksize)
The md0 device gets 8000 total IO's. Divided by 10 seconds this is 800
IOPS (So the 1MB blocksize go to the disks in 64KB blocks because the
chunksize of the md is 64KB. 800 IOPS is 16x the reported 50 IOPS
above = 1MB / 16 = 64KB)
The md0 gets 800 IOPS in 64KB blocks and this relates to 50 IOPS in 1MB blocks
Each disks gets around 800 total IO's, divided by 10 is 80 IOPS, which
is slightly lower than the number you would expect from a 7200 RPM
SATA disk (around 110 IOPS)
For the next test I introduced LVM (create a VG with md0 as PV, then
created a 2TB LV on it)
fio --name=test2 --ioengine=sync --direct=1 --rw=randread --bs=1m
--runtime=10 --filename=/dev/dm-0 --iodepth=1 --invalidate=1
read : io=705536KB, bw=70476KB/s, iops=68 , runt= 10011msec
Disk stats (read/write):
dm-0: ios=10880/0, merge=0/0, ticks=114720/0, in_queue=114784,
util=98.86%, aggrios=11024/0, aggrmerge=0/0, aggrticks=0/0,
aggrin_queue=0, aggrutil=0.00%
md0: ios=11024/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=918/0, aggrmerge=0/0, aggrticks=9684/0, aggrin_queue=9684,
aggrutil=74.05%
sdc: ios=1106/0, merge=0/0, ticks=11668/0, in_queue=11668, util=72.07%
Now the IOPS went from 50 to around 70 and the bandwidth from 50MB/s
to 70MB/s which is not what I expected since I introduced another
layer and the performance increases.
The dm-0 device gets 10880 total IO's. Divided by 10 seconds this is
1088 IOPS (So the 1MB blocksize go to the disks in 64KB blocks because
the chunksize of the md is 64KB. 1088 IOPS is 16x the reported 68 IOPS
above = 1MB / 16 = 64KB)
Each disk now does around 110 IOPS which is what you would expect for
these type of disks.
For the next test I wanted to see if i could double the performance by
striping an LV over 2 md's (so instead of using 10 disks/spindles, use
20 disks/spindles)
So i added md1 to the VG as PV.
Created a fresh LV striped across the two PV's using a 64KB stripe
size and ran the test again.
fio --name=test3 --ioengine=sync --direct=1 --rw=randread --bs=1m
--runtime=10 --filename=/dev/dm-0 --iodepth=1 --invalidate=1
Now things are getting interesting:
read : io=769024KB, bw=76849KB/s, iops=75 , runt= 10007msec
Disk stats (read/write):
dm-0: ios=190464/0, merge=0/0, ticks=1695940/0, in_queue=1695940,
util=98.03%, aggrios=96128/0, aggrmerge=0/0, aggrticks=0/0,
aggrin_queue=0, aggrutil=0.00%
md0: ios=96128/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=502/0, aggrmerge=7508/0, aggrticks=4517/0, aggrin_queue=4517,
aggrutil=55.56%
sdc: ios=583/0, merge=8745/0, ticks=5100/0, in_queue=5100, util=50.53%
md1: ios=96128/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=503/0, aggrmerge=7507/0, aggrticks=4469/0, aggrin_queue=4469,
aggrutil=55.77%
Now the total IO's in 10 seconds are 16x larger than before. 190464 /
10 = 19046,4 / 16 = 1190,4 /16 = the reported 75 IOPS above.
So the 64KB blocks seem to be split into 4KB blocks (64 / 16 = 4)
which results in a way larger total IO's.
The IO's per disk seem to be in 64KB blocks still only now with a
large MERGE figure besides it. (Now 4KB blocks are merged into 64KB
blocks?)
The performance does not double but stays the same as with 1 MD set
only the total IO's are spread among the MD's. Each disk now does
around 60 IOPS!
I still wanted to see if I could double the performance and thought it
might have something to do with LVM striping so i ditched LVM and
created a RAID0 (md6) over md0 and md1 with a chunk size of 64KB
again.
fio --name=test4 --ioengine=sync --direct=1 --rw=randread --bs=1m
--runtime=10 --filename=/dev/md6 --iodepth=1 --invalidate=1
Same story as for LVM striping only the performance is even worse:
read : io=462848KB, bw=46257KB/s, iops=45 , runt= 10006msec
md6: ios=114432/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=57856/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0,
aggrutil=0.00%
md0: ios=57856/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=302/0, aggrmerge=4519/0, aggrticks=4782/0, aggrin_queue=4782,
aggrutil=59.46%
sdc: ios=360/0, merge=5368/0, ticks=5676/0, in_queue=5676, util=55.89%
Performance is spread among the md's and each disk only does 36 IOPS now!
As a final test I wanted to know how 20 disks in a single raid6
performed so i created 1 md out of 20 disks (still using a 64KB chunk
size).
fio --name=test5 --ioengine=sync --direct=1 --rw=randread --bs=1m
--runtime=10 --filename=/dev/md0 --iodepth=1 --invalidate=1
read : io=504832KB, bw=50453KB/s, iops=49 , runt= 10006msec
md0: ios=7808/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=328/0, aggrmerge=0/0, aggrticks=4925/0, aggrin_queue=4925,
aggrutil=61.63%
sdb: ios=395/0, merge=0/0, ticks=5652/0, in_queue=5652, util=56.00%
50 IOPS total (1MB blocks) and each disk is doing around 40 IOPS (64KB blocks)
And adding LVM (VG over md0 and a 2TB LV) to that
fio --name=test6 --ioengine=sync --direct=1 --rw=randread --bs=1m
--runtime=10 --filename=/dev/md0 --iodepth=1 --invalidate=1
read : io=801792KB, bw=80163KB/s, iops=78 , runt= 10002msec
dm-0: ios=12416/0, merge=0/0, ticks=108252/0, in_queue=108252,
util=98.82%, aggrios=12528/0, aggrmerge=0/0, aggrticks=0/0,
aggrin_queue=0, aggrutil=0.00%
md0: ios=12528/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=522/0, aggrmerge=0/0, aggrticks=4547/0, aggrin_queue=4547,
aggrutil=55.80%
sdb: ios=637/0, merge=0/0, ticks=5492/0, in_queue=5492, util=54.41%
78 IOPS total (1MB blocks) and each disk is doing around 64 IOPS (64KB blocks)
I wanted to rule out the LSI controller as a bottleneck so I installed
a second LSI 9211-8i and connected 10 disks to controller 1 and 10
disks to controller 2 and created a 20 disk raid6 with these disks.
The results (test7 and test8) are pretty much the same as with using 1
controller.
So my questions are:
1) Am I overlooking/not understanding something obvious why I can't
improve performance on the system?
2) Why are the LVM tests performing better as opposed to only using MD(s)?
3) Why is the performance in test3 split between the two PV's and not
aggregated? Bottleneck somewhere, and if so how can I check which is
it?
4) Why are the IO's suddenly split into 4KB blocks when using
striping/raid0? All chunk/block/stripe sizes are 64KB.
5) Any recommendations how to improve performance with this
configuration and not limited at the performance of 10 disks?
Kind regards,
Caspar Smit
test1: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test1: (groupid=0, jobs=1): err= 0: pid=12218
read : io=518144KB, bw=51726KB/s, iops=50 , runt= 10017msec
clat (usec): min=694 , max=268167 , avg=19790.64, stdev=13161.97
lat (usec): min=694 , max=268167 , avg=19790.91, stdev=13161.97
clat percentiles (usec):
| 1.00th=[ 732], 5.00th=[11328], 10.00th=[14784], 20.00th=[16512],
| 30.00th=[17536], 40.00th=[18560], 50.00th=[19328], 60.00th=[20096],
| 70.00th=[21120], 80.00th=[22144], 90.00th=[23936], 95.00th=[25216],
| 99.00th=[56064], 99.50th=[69120], 99.90th=[268288]
bw (KB/s) : min=22304, max=66316, per=99.84%, avg=51642.68, stdev=7977.53
lat (usec) : 750=2.37%, 1000=1.78%
lat (msec) : 2=0.20%, 20=54.15%, 50=40.12%, 100=1.19%, 500=0.20%
cpu : usr=0.00%, sys=0.72%, ctx=514, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=506/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=518144KB, aggrb=51726KB/s, minb=52967KB/s, maxb=52967KB/s, mint=10017msec, maxt=10017msec
Disk stats (read/write):
md0: ios=8000/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=674/0, aggrmerge=0/0, aggrticks=10183/0, aggrin_queue=10183, aggrutil=77.97%
sdc: ios=830/0, merge=0/0, ticks=12584/0, in_queue=12584, util=63.15%
sdd: ios=818/0, merge=0/0, ticks=12480/0, in_queue=12480, util=75.60%
sde: ios=806/0, merge=0/0, ticks=13000/0, in_queue=13000, util=65.24%
sdf: ios=816/0, merge=0/0, ticks=12228/0, in_queue=12228, util=74.93%
sdg: ios=826/0, merge=0/0, ticks=12412/0, in_queue=12412, util=62.04%
sdh: ios=812/0, merge=0/0, ticks=12092/0, in_queue=12092, util=75.72%
sdi: ios=798/0, merge=0/0, ticks=11916/0, in_queue=11916, util=59.83%
sdj: ios=793/0, merge=0/0, ticks=12160/0, in_queue=12160, util=77.97%
sdk: ios=788/0, merge=0/0, ticks=11140/0, in_queue=11140, util=55.88%
sdaf: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdag: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdb: ios=809/0, merge=0/0, ticks=12188/0, in_queue=12188, util=74.93%
----
test2: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test2: (groupid=0, jobs=1): err= 0: pid=15396
read : io=705536KB, bw=70476KB/s, iops=68 , runt= 10011msec
clat (msec): min=9 , max=264 , avg=14.52, stdev=12.99
lat (msec): min=9 , max=264 , avg=14.52, stdev=12.99
clat percentiles (msec):
| 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13],
| 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 14], 60.00th=[ 15],
| 70.00th=[ 15], 80.00th=[ 16], 90.00th=[ 16], 95.00th=[ 17],
| 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 265]
bw (KB/s) : min= 4055, max=75776, per=100.00%, avg=70507.00, stdev=16129.06
lat (msec) : 10=0.58%, 20=98.55%, 50=0.58%, 250=0.15%, 500=0.15%
cpu : usr=0.00%, sys=1.28%, ctx=708, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=689/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=705536KB, aggrb=70476KB/s, minb=72167KB/s, maxb=72167KB/s, mint=10011msec, maxt=10011msec
Disk stats (read/write):
dm-0: ios=10880/0, merge=0/0, ticks=114720/0, in_queue=114784, util=98.86%, aggrios=11024/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md0: ios=11024/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=918/0, aggrmerge=0/0, aggrticks=9684/0, aggrin_queue=9684, aggrutil=74.05%
sdc: ios=1106/0, merge=0/0, ticks=11668/0, in_queue=11668, util=72.07%
sdd: ios=1100/0, merge=0/0, ticks=11700/0, in_queue=11700, util=58.90%
sde: ios=1105/0, merge=0/0, ticks=12012/0, in_queue=12012, util=74.05%
sdf: ios=1110/0, merge=0/0, ticks=11872/0, in_queue=11872, util=59.89%
sdg: ios=1119/0, merge=0/0, ticks=11256/0, in_queue=11256, util=70.29%
sdh: ios=1128/0, merge=0/0, ticks=11764/0, in_queue=11764, util=59.18%
sdi: ios=1095/0, merge=0/0, ticks=11552/0, in_queue=11552, util=73.42%
sdj: ios=1062/0, merge=0/0, ticks=11372/0, in_queue=11372, util=57.24%
sdk: ios=1087/0, merge=0/0, ticks=11344/0, in_queue=11344, util=71.08%
sdaf: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdag: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdb: ios=1112/0, merge=0/0, ticks=11676/0, in_queue=11676, util=58.58%
----
test3: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test3: (groupid=0, jobs=1): err= 0: pid=19157
read : io=769024KB, bw=76849KB/s, iops=75 , runt= 10007msec
clat (msec): min=9 , max=22 , avg=13.32, stdev= 1.60
lat (msec): min=9 , max=22 , avg=13.32, stdev= 1.60
clat percentiles (usec):
| 1.00th=[10816], 5.00th=[11328], 10.00th=[11712], 20.00th=[12224],
| 30.00th=[12608], 40.00th=[12864], 50.00th=[13120], 60.00th=[13376],
| 70.00th=[13760], 80.00th=[14016], 90.00th=[14656], 95.00th=[15296],
| 99.00th=[20608], 99.50th=[21120], 99.90th=[22400]
bw (KB/s) : min=72000, max=79395, per=100.00%, avg=76960.79, stdev=1873.61
lat (msec) : 10=0.13%, 20=98.27%, 50=1.60%
cpu : usr=0.00%, sys=3.48%, ctx=811, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=751/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=769024KB, aggrb=76848KB/s, minb=78692KB/s, maxb=78692KB/s, mint=10007msec, maxt=10007msec
Disk stats (read/write):
dm-0: ios=190464/0, merge=0/0, ticks=1695940/0, in_queue=1695940, util=98.03%, aggrios=96128/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md0: ios=96128/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=502/0, aggrmerge=7508/0, aggrticks=4517/0, aggrin_queue=4517, aggrutil=55.56%
sdc: ios=583/0, merge=8745/0, ticks=5100/0, in_queue=5100, util=50.53%
sdd: ios=604/0, merge=8996/0, ticks=5408/0, in_queue=5408, util=53.34%
sde: ios=607/0, merge=9073/0, ticks=5500/0, in_queue=5500, util=54.37%
sdf: ios=595/0, merge=8893/0, ticks=5276/0, in_queue=5276, util=52.15%
sdg: ios=609/0, merge=9087/0, ticks=5532/0, in_queue=5532, util=54.61%
sdh: ios=621/0, merge=9299/0, ticks=5612/0, in_queue=5612, util=55.56%
sdi: ios=619/0, merge=9205/0, ticks=5580/0, in_queue=5580, util=55.01%
sdj: ios=594/0, merge=8878/0, ticks=5464/0, in_queue=5464, util=53.98%
sdk: ios=597/0, merge=8939/0, ticks=5316/0, in_queue=5316, util=52.60%
sdaf: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdag: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdb: ios=601/0, merge=8983/0, ticks=5420/0, in_queue=5420, util=53.55%
md1: ios=96128/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=503/0, aggrmerge=7507/0, aggrticks=4469/0, aggrin_queue=4469, aggrutil=55.77%
sdm: ios=589/0, merge=8739/0, ticks=5320/0, in_queue=5320, util=52.40%
sdn: ios=601/0, merge=8999/0, ticks=5200/0, in_queue=5200, util=51.41%
sdo: ios=610/0, merge=9070/0, ticks=5440/0, in_queue=5440, util=53.63%
sdp: ios=593/0, merge=8895/0, ticks=5284/0, in_queue=5284, util=52.36%
sdq: ios=610/0, merge=9086/0, ticks=5320/0, in_queue=5320, util=52.40%
sdr: ios=623/0, merge=9297/0, ticks=5648/0, in_queue=5648, util=55.77%
sds: ios=618/0, merge=9206/0, ticks=5592/0, in_queue=5592, util=55.21%
sdt: ios=594/0, merge=8878/0, ticks=5204/0, in_queue=5204, util=51.45%
sdu: ios=600/0, merge=8936/0, ticks=5324/0, in_queue=5324, util=52.52%
sdah: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdai: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdl: ios=603/0, merge=8981/0, ticks=5296/0, in_queue=5296, util=52.16%
----
test4: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test4: (groupid=0, jobs=1): err= 0: pid=23762
read : io=462848KB, bw=46257KB/s, iops=45 , runt= 10006msec
clat (msec): min=11 , max=267 , avg=22.13, stdev=19.49
lat (msec): min=11 , max=267 , avg=22.13, stdev=19.49
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 19],
| 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 22],
| 70.00th=[ 23], 80.00th=[ 23], 90.00th=[ 25], 95.00th=[ 25],
| 99.00th=[ 34], 99.50th=[ 245], 99.90th=[ 269]
bw (KB/s) : min= 3923, max=54748, per=100.00%, avg=46260.32, stdev=12218.61
lat (msec) : 20=42.92%, 50=56.19%, 100=0.22%, 250=0.22%, 500=0.44%
cpu : usr=0.08%, sys=1.64%, ctx=470, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=452/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=462848KB, aggrb=46257KB/s, minb=47367KB/s, maxb=47367KB/s, mint=10006msec, maxt=10006msec
Disk stats (read/write):
md6: ios=114432/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=57856/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md0: ios=57856/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=302/0, aggrmerge=4519/0, aggrticks=4782/0, aggrin_queue=4782, aggrutil=59.46%
sdc: ios=360/0, merge=5368/0, ticks=5676/0, in_queue=5676, util=55.89%
sdd: ios=365/0, merge=5427/0, ticks=5988/0, in_queue=5988, util=58.98%
sde: ios=369/0, merge=5535/0, ticks=6000/0, in_queue=6000, util=59.46%
sdf: ios=365/0, merge=5475/0, ticks=5628/0, in_queue=5628, util=55.77%
sdg: ios=361/0, merge=5383/0, ticks=5548/0, in_queue=5548, util=54.74%
sdh: ios=357/0, merge=5323/0, ticks=5592/0, in_queue=5592, util=55.18%
sdi: ios=354/0, merge=5294/0, ticks=5736/0, in_queue=5736, util=56.69%
sdj: ios=365/0, merge=5475/0, ticks=5872/0, in_queue=5872, util=58.20%
sdk: ios=369/0, merge=5535/0, ticks=5736/0, in_queue=5736, util=56.85%
sdaf: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdag: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdb: ios=362/0, merge=5414/0, ticks=5616/0, in_queue=5616, util=55.42%
md1: ios=57856/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=301/0, aggrmerge=4519/0, aggrticks=4770/0, aggrin_queue=4770, aggrutil=59.15%
sdm: ios=359/0, merge=5369/0, ticks=5708/0, in_queue=5708, util=56.45%
sdn: ios=362/0, merge=5430/0, ticks=5668/0, in_queue=5668, util=56.17%
sdo: ios=371/0, merge=5533/0, ticks=6000/0, in_queue=6000, util=59.15%
sdp: ios=366/0, merge=5474/0, ticks=5900/0, in_queue=5900, util=58.35%
sdq: ios=359/0, merge=5385/0, ticks=5804/0, in_queue=5804, util=57.52%
sdr: ios=355/0, merge=5325/0, ticks=5492/0, in_queue=5492, util=54.43%
sds: ios=353/0, merge=5295/0, ticks=5436/0, in_queue=5436, util=53.88%
sdt: ios=365/0, merge=5475/0, ticks=5620/0, in_queue=5620, util=55.70%
sdu: ios=370/0, merge=5534/0, ticks=5700/0, in_queue=5700, util=56.29%
sdah: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdai: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdl: ios=362/0, merge=5414/0, ticks=5920/0, in_queue=5920, util=58.55%
----
test5: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test5: (groupid=0, jobs=1): err= 0: pid=32210
read : io=504832KB, bw=50453KB/s, iops=49 , runt= 10006msec
clat (msec): min=11 , max=60 , avg=20.29, stdev= 3.76
lat (msec): min=11 , max=60 , avg=20.29, stdev= 3.76
clat percentiles (usec):
| 1.00th=[13248], 5.00th=[15680], 10.00th=[16768], 20.00th=[18048],
| 30.00th=[18816], 40.00th=[19584], 50.00th=[20096], 60.00th=[20864],
| 70.00th=[21632], 80.00th=[22400], 90.00th=[23424], 95.00th=[24704],
| 99.00th=[25984], 99.50th=[49920], 99.90th=[60160]
bw (KB/s) : min=39922, max=54425, per=99.82%, avg=50360.74, stdev=3005.20
lat (msec) : 20=48.68%, 50=50.71%, 100=0.61%
cpu : usr=0.00%, sys=0.76%, ctx=499, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=493/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=504832KB, aggrb=50452KB/s, minb=51663KB/s, maxb=51663KB/s, mint=10006msec, maxt=10006msec
Disk stats (read/write):
md0: ios=7808/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=328/0, aggrmerge=0/0, aggrticks=4925/0, aggrin_queue=4925, aggrutil=61.63%
sdb: ios=395/0, merge=0/0, ticks=5652/0, in_queue=5652, util=56.00%
sdc: ios=404/0, merge=0/0, ticks=5932/0, in_queue=5932, util=58.78%
sdd: ios=390/0, merge=0/0, ticks=5856/0, in_queue=5856, util=58.03%
sde: ios=397/0, merge=0/0, ticks=6180/0, in_queue=6180, util=61.24%
sdf: ios=396/0, merge=0/0, ticks=5940/0, in_queue=5940, util=58.86%
sdg: ios=392/0, merge=0/0, ticks=5772/0, in_queue=5772, util=57.19%
sdh: ios=385/0, merge=0/0, ticks=5600/0, in_queue=5600, util=55.49%
sdi: ios=377/0, merge=0/0, ticks=5800/0, in_queue=5800, util=57.47%
sdj: ios=378/0, merge=0/0, ticks=5768/0, in_queue=5768, util=57.15%
sdk: ios=394/0, merge=0/0, ticks=5820/0, in_queue=5820, util=57.67%
sdl: ios=398/0, merge=0/0, ticks=6108/0, in_queue=6108, util=60.52%
sdm: ios=390/0, merge=0/0, ticks=5856/0, in_queue=5856, util=58.03%
sdn: ios=402/0, merge=0/0, ticks=6000/0, in_queue=6000, util=59.45%
sdo: ios=400/0, merge=0/0, ticks=6220/0, in_queue=6220, util=61.63%
sdp: ios=398/0, merge=0/0, ticks=6068/0, in_queue=6068, util=60.13%
sdq: ios=401/0, merge=0/0, ticks=6200/0, in_queue=6200, util=61.44%
sdr: ios=401/0, merge=0/0, ticks=5832/0, in_queue=5832, util=57.79%
sds: ios=392/0, merge=0/0, ticks=5820/0, in_queue=5820, util=57.68%
sdt: ios=401/0, merge=0/0, ticks=5908/0, in_queue=5908, util=58.55%
sdu: ios=397/0, merge=0/0, ticks=5888/0, in_queue=5888, util=58.35%
sdaf: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdag: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdah: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdai: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
----
test6: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test6: (groupid=0, jobs=1): err= 0: pid=1850
read : io=801792KB, bw=80163KB/s, iops=78 , runt= 10002msec
clat (msec): min=9 , max=60 , avg=12.77, stdev= 2.77
lat (msec): min=9 , max=60 , avg=12.77, stdev= 2.77
clat percentiles (usec):
| 1.00th=[10048], 5.00th=[10944], 10.00th=[11328], 20.00th=[11712],
| 30.00th=[12096], 40.00th=[12352], 50.00th=[12608], 60.00th=[12864],
| 70.00th=[13120], 80.00th=[13376], 90.00th=[13888], 95.00th=[14144],
| 99.00th=[19840], 99.50th=[28032], 99.90th=[60672]
bw (KB/s) : min=64631, max=84329, per=100.00%, avg=80169.58, stdev=4137.01
lat (msec) : 10=0.89%, 20=98.21%, 50=0.64%, 100=0.26%
cpu : usr=0.04%, sys=1.36%, ctx=802, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=783/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=801792KB, aggrb=80163KB/s, minb=82087KB/s, maxb=82087KB/s, mint=10002msec, maxt=10002msec
Disk stats (read/write):
dm-0: ios=12416/0, merge=0/0, ticks=108252/0, in_queue=108252, util=98.82%, aggrios=12528/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md0: ios=12528/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=522/0, aggrmerge=0/0, aggrticks=4547/0, aggrin_queue=4547, aggrutil=55.80%
sdb: ios=637/0, merge=0/0, ticks=5492/0, in_queue=5492, util=54.41%
sdc: ios=634/0, merge=0/0, ticks=5320/0, in_queue=5320, util=52.71%
sdd: ios=631/0, merge=0/0, ticks=5588/0, in_queue=5588, util=55.37%
sde: ios=629/0, merge=0/0, ticks=5584/0, in_queue=5584, util=55.33%
sdf: ios=620/0, merge=0/0, ticks=5552/0, in_queue=5552, util=55.01%
sdg: ios=610/0, merge=0/0, ticks=5420/0, in_queue=5420, util=53.70%
sdh: ios=631/0, merge=0/0, ticks=5468/0, in_queue=5468, util=54.18%
sdi: ios=619/0, merge=0/0, ticks=5416/0, in_queue=5416, util=53.66%
sdj: ios=630/0, merge=0/0, ticks=5424/0, in_queue=5424, util=53.74%
sdk: ios=630/0, merge=0/0, ticks=5564/0, in_queue=5564, util=55.13%
sdl: ios=607/0, merge=0/0, ticks=5200/0, in_queue=5200, util=51.52%
sdm: ios=637/0, merge=0/0, ticks=5544/0, in_queue=5544, util=54.93%
sdn: ios=619/0, merge=0/0, ticks=5416/0, in_queue=5416, util=53.66%
sdo: ios=640/0, merge=0/0, ticks=5568/0, in_queue=5568, util=55.17%
sdp: ios=636/0, merge=0/0, ticks=5632/0, in_queue=5632, util=55.80%
sdq: ios=615/0, merge=0/0, ticks=5376/0, in_queue=5376, util=53.26%
sdr: ios=622/0, merge=0/0, ticks=5284/0, in_queue=5284, util=52.35%
sds: ios=618/0, merge=0/0, ticks=5428/0, in_queue=5428, util=53.78%
sdt: ios=631/0, merge=0/0, ticks=5528/0, in_queue=5528, util=54.78%
sdu: ios=632/0, merge=0/0, ticks=5340/0, in_queue=5340, util=52.91%
sdaf: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdag: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdah: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdai: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
----
test7: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test7: (groupid=0, jobs=1): err= 0: pid=13783
read : io=505856KB, bw=50545KB/s, iops=49 , runt= 10008msec
clat (msec): min=11 , max=31 , avg=20.25, stdev= 2.89
lat (msec): min=11 , max=31 , avg=20.25, stdev= 2.89
clat percentiles (usec):
| 1.00th=[13760], 5.00th=[15552], 10.00th=[16768], 20.00th=[17792],
| 30.00th=[18560], 40.00th=[19328], 50.00th=[20096], 60.00th=[20864],
| 70.00th=[21888], 80.00th=[22912], 90.00th=[23936], 95.00th=[24960],
| 99.00th=[26240], 99.50th=[27776], 99.90th=[31616]
bw (KB/s) : min=48282, max=52965, per=99.93%, avg=50507.63, stdev=1511.43
lat (msec) : 20=47.37%, 50=52.63%
cpu : usr=0.00%, sys=0.72%, ctx=501, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=494/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=505856KB, aggrb=50545KB/s, minb=51758KB/s, maxb=51758KB/s, mint=10008msec, maxt=10008msec
Disk stats (read/write):
md0: ios=7824/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=395/0, aggrmerge=0/0, aggrticks=5965/0, aggrin_queue=5965, aggrutil=61.25%
sdb: ios=396/0, merge=0/0, ticks=5732/0, in_queue=5732, util=56.77%
sdba: ios=405/0, merge=0/0, ticks=6124/0, in_queue=6124, util=60.65%
sdbb: ios=391/0, merge=0/0, ticks=6016/0, in_queue=6016, util=59.58%
sdbc: ios=398/0, merge=0/0, ticks=6184/0, in_queue=6184, util=61.25%
sdbd: ios=397/0, merge=0/0, ticks=5928/0, in_queue=5928, util=58.72%
sdbg: ios=393/0, merge=0/0, ticks=6084/0, in_queue=6084, util=60.26%
sdbh: ios=386/0, merge=0/0, ticks=5884/0, in_queue=5884, util=58.28%
sdbi: ios=378/0, merge=0/0, ticks=5856/0, in_queue=5856, util=58.00%
sdbl: ios=379/0, merge=0/0, ticks=5948/0, in_queue=5948, util=58.91%
sdbr: ios=394/0, merge=0/0, ticks=5988/0, in_queue=5988, util=59.31%
sdbs: ios=398/0, merge=0/0, ticks=6072/0, in_queue=6072, util=60.14%
sdc: ios=390/0, merge=0/0, ticks=5712/0, in_queue=5712, util=56.58%
sdd: ios=402/0, merge=0/0, ticks=5912/0, in_queue=5912, util=58.56%
sde: ios=401/0, merge=0/0, ticks=6140/0, in_queue=6140, util=60.82%
sdf: ios=399/0, merge=0/0, ticks=5844/0, in_queue=5844, util=57.88%
sdg: ios=402/0, merge=0/0, ticks=6032/0, in_queue=6032, util=59.75%
sdh: ios=402/0, merge=0/0, ticks=5976/0, in_queue=5976, util=59.19%
sdi: ios=393/0, merge=0/0, ticks=5952/0, in_queue=5952, util=58.95%
sdj: ios=402/0, merge=0/0, ticks=6088/0, in_queue=6088, util=60.30%
sdk: ios=398/0, merge=0/0, ticks=5828/0, in_queue=5828, util=57.73%
----
test8: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.4
Starting 1 process
test8: (groupid=0, jobs=1): err= 0: pid=14153
read : io=707584KB, bw=70751KB/s, iops=69 , runt= 10001msec
clat (msec): min=8 , max=607 , avg=14.47, stdev=31.84
lat (msec): min=8 , max=607 , avg=14.47, stdev=31.84
clat percentiles (msec):
| 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 13], 60.00th=[ 13],
| 70.00th=[ 14], 80.00th=[ 14], 90.00th=[ 15], 95.00th=[ 15],
| 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 611]
bw (KB/s) : min= 1686, max=82483, per=100.00%, avg=71972.26, stdev=24807.51
lat (msec) : 10=0.29%, 20=98.84%, 50=0.58%, 750=0.29%
cpu : usr=0.04%, sys=1.20%, ctx=711, majf=0, minf=284
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=691/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=707584KB, aggrb=70751KB/s, minb=72449KB/s, maxb=72449KB/s, mint=10001msec, maxt=10001msec
Disk stats (read/write):
dm-0: ios=10944/0, merge=0/0, ticks=101672/0, in_queue=101712, util=98.78%, aggrios=11056/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md0: ios=11056/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=552/0, aggrmerge=0/0, aggrticks=5137/0, aggrin_queue=5137, aggrutil=54.71%
sdb: ios=565/0, merge=0/0, ticks=4988/0, in_queue=4988, util=49.41%
sdba: ios=560/0, merge=0/0, ticks=5004/0, in_queue=5004, util=49.56%
sdbb: ios=554/0, merge=0/0, ticks=5488/0, in_queue=5488, util=54.36%
sdbc: ios=552/0, merge=0/0, ticks=5524/0, in_queue=5524, util=54.71%
sdbd: ios=545/0, merge=0/0, ticks=4744/0, in_queue=4744, util=46.99%
sdbg: ios=535/0, merge=0/0, ticks=5376/0, in_queue=5376, util=53.25%
sdbh: ios=558/0, merge=0/0, ticks=4960/0, in_queue=4960, util=49.13%
sdbi: ios=542/0, merge=0/0, ticks=4764/0, in_queue=4764, util=47.19%
sdbl: ios=556/0, merge=0/0, ticks=5480/0, in_queue=5480, util=54.28%
sdbr: ios=555/0, merge=0/0, ticks=5452/0, in_queue=5452, util=54.01%
sdbs: ios=540/0, merge=0/0, ticks=5408/0, in_queue=5408, util=53.57%
sdc: ios=569/0, merge=0/0, ticks=5340/0, in_queue=5340, util=52.90%
sdd: ios=549/0, merge=0/0, ticks=4888/0, in_queue=4888, util=48.42%
sde: ios=568/0, merge=0/0, ticks=5016/0, in_queue=5016, util=49.69%
sdf: ios=560/0, merge=0/0, ticks=5032/0, in_queue=5032, util=49.85%
sdg: ios=544/0, merge=0/0, ticks=4744/0, in_queue=4744, util=46.99%
sdh: ios=547/0, merge=0/0, ticks=4860/0, in_queue=4860, util=48.14%
sdi: ios=546/0, merge=0/0, ticks=5320/0, in_queue=5320, util=52.70%
sdj: ios=554/0, merge=0/0, ticks=5452/0, in_queue=5452, util=54.01%
sdk: ios=557/0, merge=0/0, ticks=4912/0, in_queue=4912, util=48.66%
