Carrying out some scalability tests on high I/O systems. More tests to
come using the Phoronix Test Suite.
Fio summary:
24 jobs
Direct IO
Randwrite test
Total of about 80k IOPs at 3.5k IOPS per thread.
Test rig specs:
2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (12 physical cores, 24 logical)
12x 2TB Seagate nearline-SAS in RAID6 on LSI Logic / Symbios Logic LSI
MegaSAS 9260
2x Intel 520 SSDs 120GB in RAID0
The 2 520s are striped with md raid as /dev/md0 whch is formatted as a
bcache cache device using 1M buckets and 8k hard block size
Backing device is the big old raid6.
Random IO performance of the native raid0 is about 96K IOPs, backing
device in the realm of 1600ish.
However the backing device has a sequential IO performance of about 1.5GB/s.
Below are some quick findings using fio - showing very good
scalability of bcache even with very very fast SSDs.
*Note: The SSDs are connected via a SATA2 interface being somewhat of
a bottleneck.
dev05 ~ # fio --filename=/dev/bcache0 --direct=1 --rw=randwrite
--bs=4096 --size=10001078620 --runtime=60 --name=file1 --iodepth=32
--numjobs=24 --ioengine=libaio
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio 2.0.3
Starting 24 processes
Jobs: 10 (f=10): [______ww_wwww__E__www_w_] [7.8% done] [0K/322.4M /s]
[0 /78.7K iops] [eta 12m:15s]
file1: (groupid=0, jobs=1): err= 0: pid=3993
write: io=821604KB, bw=13645KB/s, iops=3411 , runt= 60214msec
slat (usec): min=4 , max=113487 , avg=114.47, stdev=2376.82
clat (usec): min=460 , max=1617.5K, avg=9223.06, stdev=19736.65
lat (usec): min=476 , max=1617.5K, avg=9337.33, stdev=20514.45
clat percentiles (usec):
| 1.00th=[ 1288], 5.00th=[ 1896], 10.00th=[ 2416], 20.00th=[ 3120],
| 30.00th=[ 3728], 40.00th=[ 4384], 50.00th=[ 5088], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9152], 95.00th=[37120],
| 99.00th=[94720], 99.50th=[138240], 99.90th=[216064]
bw (KB/s) : min= 5769, max=50211, per=4.33%, avg=13718.03, stdev=5147.65
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.16%
lat (msec) : 2=5.69%, 4=28.76%, 10=56.24%, 20=1.62%, 50=3.89%
lat (msec) : 100=2.72%, 250=0.87%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=3.05%, ctx=39039, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=205401/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3994
write: io=773564KB, bw=12864KB/s, iops=3216 , runt= 60134msec
slat (usec): min=4 , max=121438 , avg=123.82, stdev=2487.05
clat (usec): min=473 , max=1676.1K, avg=9799.12, stdev=22553.15
lat (usec): min=489 , max=1760.2K, avg=9922.61, stdev=23345.78
clat percentiles (usec):
| 1.00th=[ 1448], 5.00th=[ 2160], 10.00th=[ 2672], 20.00th=[ 3280],
| 30.00th=[ 3856], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9408], 95.00th=[38656],
| 99.00th=[99840], 99.50th=[156672], 99.90th=[228352]
bw (KB/s) : min= 5158, max=37016, per=4.07%, avg=12898.20, stdev=4203.06
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.12%
lat (msec) : 2=3.54%, 4=28.30%, 10=58.52%, 20=1.64%, 50=4.03%
lat (msec) : 100=2.84%, 250=0.95%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.89%, sys=2.95%, ctx=38309, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=193391/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3995
write: io=783660KB, bw=13060KB/s, iops=3265 , runt= 60004msec
slat (usec): min=4 , max=118055 , avg=120.56, stdev=2423.94
clat (usec): min=530 , max=1082.8K, avg=9675.59, stdev=20228.23
lat (usec): min=551 , max=1142.8K, avg=9796.35, stdev=21045.95
clat percentiles (usec):
| 1.00th=[ 1480], 5.00th=[ 2256], 10.00th=[ 2736], 20.00th=[ 3408],
| 30.00th=[ 3952], 40.00th=[ 4640], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9536], 95.00th=[38144],
| 99.00th=[94720], 99.50th=[150528], 99.90th=[222208]
bw (KB/s) : min= 4756, max=21700, per=4.11%, avg=13022.62, stdev=3325.00
lat (usec) : 750=0.01%, 1000=0.10%
lat (msec) : 2=3.14%, 4=27.40%, 10=59.65%, 20=1.81%, 50=4.03%
lat (msec) : 100=2.92%, 250=0.90%, 500=0.03%, 750=0.01%, 2000=0.01%
cpu : usr=0.91%, sys=3.06%, ctx=38590, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=195915/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3996
write: io=799344KB, bw=13233KB/s, iops=3308 , runt= 60405msec
slat (usec): min=4 , max=113409 , avg=115.84, stdev=2365.77
clat (usec): min=648 , max=1313.9K, avg=9487.45, stdev=21360.00
lat (usec): min=661 , max=1313.9K, avg=9603.37, stdev=22116.87
clat percentiles (usec):
| 1.00th=[ 1464], 5.00th=[ 2096], 10.00th=[ 2544], 20.00th=[ 3184],
| 30.00th=[ 3792], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5792],
| 70.00th=[ 6432], 80.00th=[ 7328], 90.00th=[ 9280], 95.00th=[37120],
| 99.00th=[96768], 99.50th=[152576], 99.90th=[222208]
bw (KB/s) : min= 5483, max=34091, per=4.20%, avg=13305.74, stdev=3850.52
lat (usec) : 750=0.01%, 1000=0.07%
lat (msec) : 2=4.13%, 4=29.36%, 10=57.07%, 20=1.78%, 50=4.05%
lat (msec) : 100=2.56%, 250=0.93%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=2.95%, ctx=38593, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=199836/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3997
write: io=821316KB, bw=13643KB/s, iops=3410 , runt= 60200msec
slat (usec): min=4 , max=117990 , avg=110.12, stdev=2280.98
clat (usec): min=568 , max=1796.8K, avg=9237.17, stdev=20452.93
lat (usec): min=585 , max=1796.8K, avg=9347.04, stdev=21191.36
clat percentiles (usec):
| 1.00th=[ 1416], 5.00th=[ 2128], 10.00th=[ 2544], 20.00th=[ 3152],
| 30.00th=[ 3728], 40.00th=[ 4384], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[36608],
| 99.00th=[94720], 99.50th=[148480], 99.90th=[214016]
bw (KB/s) : min= 5928, max=43330, per=4.31%, avg=13668.87, stdev=4570.80
lat (usec) : 750=0.01%, 1000=0.17%
lat (msec) : 2=3.79%, 4=30.50%, 10=56.62%, 20=1.52%, 50=4.01%
lat (msec) : 100=2.45%, 250=0.90%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.97%, sys=3.02%, ctx=39689, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=205329/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3999
write: io=849880KB, bw=14154KB/s, iops=3538 , runt= 60047msec
slat (usec): min=4 , max=123486 , avg=105.83, stdev=2216.28
clat (usec): min=446 , max=1383.1K, avg=8926.80, stdev=19692.96
lat (usec): min=461 , max=1384.2K, avg=9032.43, stdev=20445.23
clat percentiles (usec):
| 1.00th=[ 1368], 5.00th=[ 1944], 10.00th=[ 2352], 20.00th=[ 3024],
| 30.00th=[ 3632], 40.00th=[ 4256], 50.00th=[ 5024], 60.00th=[ 5664],
| 70.00th=[ 6368], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[33536],
| 99.00th=[91648], 99.50th=[146432], 99.90th=[220160]
bw (KB/s) : min= 5807, max=48232, per=4.48%, avg=14192.07, stdev=5047.52
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.18%
lat (msec) : 2=5.43%, 4=30.44%, 10=55.07%, 20=1.70%, 50=3.99%
lat (msec) : 100=2.27%, 250=0.86%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=1.03%, sys=3.21%, ctx=40302, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=212470/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4000
write: io=773688KB, bw=12894KB/s, iops=3223 , runt= 60003msec
slat (usec): min=4 , max=126488 , avg=119.41, stdev=2415.30
clat (usec): min=694 , max=949429 , avg=9803.35, stdev=20713.09
lat (usec): min=721 , max=949436 , avg=9922.96, stdev=21497.37
clat percentiles (usec):
| 1.00th=[ 1496], 5.00th=[ 2256], 10.00th=[ 2736], 20.00th=[ 3408],
| 30.00th=[ 3984], 40.00th=[ 4640], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9536], 95.00th=[38656],
| 99.00th=[104960], 99.50th=[154624], 99.90th=[228352]
bw (KB/s) : min= 5880, max=19437, per=4.04%, avg=12796.90, stdev=3142.95
lat (usec) : 750=0.01%, 1000=0.09%
lat (msec) : 2=2.99%, 4=26.97%, 10=60.34%, 20=1.76%, 50=4.06%
lat (msec) : 100=2.70%, 250=1.04%, 500=0.04%, 750=0.01%, 1000=0.01%
cpu : usr=0.89%, sys=2.93%, ctx=39334, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=193422/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4001
write: io=769892KB, bw=12831KB/s, iops=3207 , runt= 60004msec
slat (usec): min=4 , max=128937 , avg=125.40, stdev=2497.02
clat (usec): min=543 , max=1641.2K, avg=9845.70, stdev=21517.57
lat (usec): min=557 , max=1641.2K, avg=9971.31, stdev=22291.20
clat percentiles (usec):
| 1.00th=[ 1432], 5.00th=[ 2096], 10.00th=[ 2576], 20.00th=[ 3280],
| 30.00th=[ 3888], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9920], 95.00th=[38144],
| 99.00th=[104960], 99.50th=[154624], 99.90th=[232448]
bw (KB/s) : min= 6854, max=23338, per=4.03%, avg=12774.62, stdev=3244.76
lat (usec) : 750=0.01%, 1000=0.17%
lat (msec) : 2=3.97%, 4=27.74%, 10=58.14%, 20=1.71%, 50=4.53%
lat (msec) : 100=2.68%, 250=1.00%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.89%, sys=3.00%, ctx=38235, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=192473/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4002
write: io=828440KB, bw=13807KB/s, iops=3451 , runt= 60003msec
slat (usec): min=4 , max=113456 , avg=107.59, stdev=2251.36
clat (usec): min=717 , max=1012.1K, avg=9159.59, stdev=19972.65
lat (usec): min=731 , max=1013.3K, avg=9267.38, stdev=20704.40
clat percentiles (usec):
| 1.00th=[ 1416], 5.00th=[ 2024], 10.00th=[ 2480], 20.00th=[ 3120],
| 30.00th=[ 3696], 40.00th=[ 4384], 50.00th=[ 5088], 60.00th=[ 5664],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[35584],
| 99.00th=[98816], 99.50th=[154624], 99.90th=[226304]
bw (KB/s) : min= 4280, max=25462, per=4.35%, avg=13769.89, stdev=4158.74
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=4.67%, 4=29.90%, 10=56.57%, 20=1.58%, 50=3.82%
lat (msec) : 100=2.35%, 250=0.94%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.99%, sys=3.13%, ctx=40961, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=207110/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4003
write: io=788452KB, bw=13090KB/s, iops=3272 , runt= 60231msec
slat (usec): min=4 , max=135424 , avg=119.03, stdev=2426.72
clat (usec): min=514 , max=1681.9K, avg=9614.94, stdev=21243.36
lat (usec): min=527 , max=1681.9K, avg=9734.19, stdev=22016.46
clat percentiles (usec):
| 1.00th=[ 1336], 5.00th=[ 2064], 10.00th=[ 2512], 20.00th=[ 3216],
| 30.00th=[ 3792], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7328], 90.00th=[ 9280], 95.00th=[38144],
| 99.00th=[104960], 99.50th=[158720], 99.90th=[228352]
bw (KB/s) : min= 4976, max=27612, per=4.13%, avg=13082.37, stdev=3858.63
lat (usec) : 750=0.01%, 1000=0.14%
lat (msec) : 2=4.44%, 4=28.95%, 10=57.10%, 20=1.67%, 50=3.93%
lat (msec) : 100=2.66%, 250=1.04%, 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.96%, sys=3.04%, ctx=39625, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=197113/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4004
write: io=790536KB, bw=13175KB/s, iops=3293 , runt= 60002msec
slat (usec): min=4 , max=114963 , avg=119.88, stdev=2419.63
clat (usec): min=537 , max=1023.4K, avg=9591.03, stdev=20883.64
lat (usec): min=552 , max=1023.4K, avg=9711.11, stdev=21661.57
clat percentiles (usec):
| 1.00th=[ 1400], 5.00th=[ 2024], 10.00th=[ 2480], 20.00th=[ 3216],
| 30.00th=[ 3824], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9280], 95.00th=[38144],
| 99.00th=[101888], 99.50th=[158720], 99.90th=[228352]
bw (KB/s) : min= 4088, max=30406, per=4.13%, avg=13071.40, stdev=4094.62
lat (usec) : 750=0.01%, 1000=0.13%
lat (msec) : 2=4.63%, 4=28.15%, 10=57.73%, 20=1.64%, 50=4.03%
lat (msec) : 100=2.63%, 250=0.99%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.97%, sys=2.95%, ctx=39111, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=197634/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4005
write: io=877548KB, bw=14626KB/s, iops=3656 , runt= 60001msec
slat (usec): min=4 , max=113739 , avg=101.26, stdev=2181.30
clat (usec): min=500 , max=1509.9K, avg=8646.60, stdev=18853.03
lat (usec): min=517 , max=1509.9K, avg=8748.06, stdev=19575.58
clat percentiles (usec):
| 1.00th=[ 1272], 5.00th=[ 1848], 10.00th=[ 2288], 20.00th=[ 2992],
| 30.00th=[ 3568], 40.00th=[ 4192], 50.00th=[ 4896], 60.00th=[ 5536],
| 70.00th=[ 6240], 80.00th=[ 7136], 90.00th=[ 8640], 95.00th=[34048],
| 99.00th=[87552], 99.50th=[132096], 99.90th=[218112]
bw (KB/s) : min= 6671, max=30083, per=4.54%, avg=14369.59, stdev=4168.17
lat (usec) : 750=0.01%, 1000=0.16%
lat (msec) : 2=6.39%, 4=30.90%, 10=54.36%, 20=1.32%, 50=3.77%
lat (msec) : 100=2.31%, 250=0.73%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.99%, sys=3.31%, ctx=40531, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=219387/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4006
write: io=770940KB, bw=12849KB/s, iops=3212 , runt= 60002msec
slat (usec): min=4 , max=117996 , avg=124.61, stdev=2459.97
clat (usec): min=446 , max=1537.3K, avg=9834.30, stdev=22022.92
lat (usec): min=458 , max=1537.7K, avg=9959.11, stdev=22800.07
clat percentiles (usec):
| 1.00th=[ 1304], 5.00th=[ 1880], 10.00th=[ 2320], 20.00th=[ 3088],
| 30.00th=[ 3728], 40.00th=[ 4384], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6496], 80.00th=[ 7392], 90.00th=[ 9792], 95.00th=[39168],
| 99.00th=[115200], 99.50th=[164864], 99.90th=[228352]
bw (KB/s) : min= 3659, max=40632, per=4.00%, avg=12683.36, stdev=4691.41
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.21%
lat (msec) : 2=6.05%, 4=28.06%, 10=55.83%, 20=1.69%, 50=4.23%
lat (msec) : 100=2.72%, 250=1.12%, 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.93%, sys=2.93%, ctx=38130, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=192735/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4007
write: io=784360KB, bw=13005KB/s, iops=3251 , runt= 60314msec
slat (usec): min=4 , max=128950 , avg=119.95, stdev=2394.53
clat (usec): min=527 , max=1082.5K, avg=9666.92, stdev=20835.35
lat (usec): min=553 , max=1082.5K, avg=9787.07, stdev=21616.62
clat percentiles (usec):
| 1.00th=[ 1368], 5.00th=[ 2040], 10.00th=[ 2480], 20.00th=[ 3248],
| 30.00th=[ 3856], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5856],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9536], 95.00th=[37632],
| 99.00th=[101888], 99.50th=[156672], 99.90th=[226304]
bw (KB/s) : min= 4433, max=26743, per=4.13%, avg=13096.11, stdev=3884.65
lat (usec) : 750=0.01%, 1000=0.17%
lat (msec) : 2=4.41%, 4=27.75%, 10=58.12%, 20=1.63%, 50=4.21%
lat (msec) : 100=2.66%, 250=0.99%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.89%, sys=3.05%, ctx=39152, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196090/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4008
write: io=784736KB, bw=13061KB/s, iops=3265 , runt= 60084msec
slat (usec): min=4 , max=112108 , avg=121.25, stdev=2421.10
clat (usec): min=583 , max=1580.3K, avg=9660.47, stdev=21404.93
lat (usec): min=595 , max=1603.4K, avg=9781.91, stdev=22175.06
clat percentiles (usec):
| 1.00th=[ 1464], 5.00th=[ 2064], 10.00th=[ 2512], 20.00th=[ 3216],
| 30.00th=[ 3856], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6496], 80.00th=[ 7392], 90.00th=[ 9408], 95.00th=[38144],
| 99.00th=[95744], 99.50th=[152576], 99.90th=[226304]
bw (KB/s) : min= 6019, max=31992, per=4.13%, avg=13095.62, stdev=4157.33
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=4.19%, 4=27.99%, 10=58.12%, 20=1.54%, 50=4.28%
lat (msec) : 100=2.79%, 250=0.90%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=2.99%, ctx=38544, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196184/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4010
write: io=815684KB, bw=13594KB/s, iops=3398 , runt= 60003msec
slat (usec): min=4 , max=141599 , avg=111.80, stdev=2314.86
clat (usec): min=600 , max=1217.5K, avg=9300.28, stdev=20275.17
lat (usec): min=610 , max=1217.5K, avg=9412.28, stdev=21008.51
clat percentiles (usec):
| 1.00th=[ 1480], 5.00th=[ 2128], 10.00th=[ 2544], 20.00th=[ 3152],
| 30.00th=[ 3760], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 8896], 95.00th=[36096],
| 99.00th=[97792], 99.50th=[152576], 99.90th=[222208]
bw (KB/s) : min= 5160, max=24846, per=4.29%, avg=13584.51, stdev=3625.42
lat (usec) : 750=0.01%, 1000=0.09%
lat (msec) : 2=3.81%, 4=29.84%, 10=57.34%, 20=1.53%, 50=3.84%
lat (msec) : 100=2.57%, 250=0.93%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.92%, sys=3.20%, ctx=39886, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=203921/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4011
write: io=784100KB, bw=13055KB/s, iops=3263 , runt= 60062msec
slat (usec): min=4 , max=99300 , avg=116.71, stdev=2345.60
clat (usec): min=591 , max=838633 , avg=9673.18, stdev=21239.11
lat (usec): min=605 , max=838655 , avg=9789.84, stdev=21997.58
clat percentiles (usec):
| 1.00th=[ 1384], 5.00th=[ 1960], 10.00th=[ 2448], 20.00th=[ 3216],
| 30.00th=[ 3792], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9408], 95.00th=[37120],
| 99.00th=[108032], 99.50th=[162816], 99.90th=[230400]
bw (KB/s) : min= 5988, max=28728, per=4.13%, avg=13098.09, stdev=3959.17
lat (usec) : 750=0.01%, 1000=0.11%
lat (msec) : 2=5.19%, 4=27.85%, 10=57.38%, 20=1.66%, 50=4.18%
lat (msec) : 100=2.52%, 250=1.05%, 500=0.04%, 750=0.01%, 1000=0.01%
cpu : usr=0.97%, sys=2.94%, ctx=40022, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196025/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4012
write: io=784720KB, bw=13078KB/s, iops=3269 , runt= 60005msec
slat (usec): min=4 , max=112089 , avg=118.79, stdev=2392.98
clat (usec): min=451 , max=1475.5K, avg=9663.55, stdev=21206.38
lat (usec): min=464 , max=1475.6K, avg=9782.54, stdev=21983.94
clat percentiles (usec):
| 1.00th=[ 1288], 5.00th=[ 1992], 10.00th=[ 2480], 20.00th=[ 3248],
| 30.00th=[ 3888], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6496], 80.00th=[ 7392], 90.00th=[ 9408], 95.00th=[37120],
| 99.00th=[100864], 99.50th=[158720], 99.90th=[224256]
bw (KB/s) : min= 5946, max=23496, per=4.12%, avg=13059.17, stdev=3462.91
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.18%
lat (msec) : 2=4.84%, 4=26.82%, 10=58.59%, 20=1.76%, 50=4.07%
lat (msec) : 100=2.70%, 250=0.98%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.95%, sys=2.99%, ctx=39607, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196180/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4013
write: io=793884KB, bw=13231KB/s, iops=3307 , runt= 60003msec
slat (usec): min=4 , max=115547 , avg=116.91, stdev=2359.21
clat (usec): min=628 , max=1535.7K, avg=9553.22, stdev=20747.17
lat (usec): min=634 , max=1535.8K, avg=9670.33, stdev=21533.58
clat percentiles (usec):
| 1.00th=[ 1512], 5.00th=[ 2160], 10.00th=[ 2640], 20.00th=[ 3280],
| 30.00th=[ 3856], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9536], 95.00th=[37632],
| 99.00th=[99840], 99.50th=[152576], 99.90th=[224256]
bw (KB/s) : min= 6772, max=27640, per=4.15%, avg=13157.26, stdev=3657.51
lat (usec) : 750=0.01%, 1000=0.07%
lat (msec) : 2=3.52%, 4=28.95%, 10=57.77%, 20=1.86%, 50=4.24%
lat (msec) : 100=2.57%, 250=0.96%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.95%, sys=2.97%, ctx=38933, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=198471/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4014
write: io=826984KB, bw=13782KB/s, iops=3445 , runt= 60003msec
slat (usec): min=4 , max=126500 , avg=110.53, stdev=2316.49
clat (usec): min=605 , max=1631.9K, avg=9172.40, stdev=20560.58
lat (usec): min=619 , max=1653.4K, avg=9283.13, stdev=21293.23
clat percentiles (usec):
| 1.00th=[ 1352], 5.00th=[ 2040], 10.00th=[ 2480], 20.00th=[ 3120],
| 30.00th=[ 3696], 40.00th=[ 4320], 50.00th=[ 5024], 60.00th=[ 5664],
| 70.00th=[ 6368], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[36096],
| 99.00th=[93696], 99.50th=[148480], 99.90th=[222208]
bw (KB/s) : min= 4413, max=27808, per=4.28%, avg=13551.74, stdev=3870.91
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=4.57%, 4=30.57%, 10=55.76%, 20=1.70%, 50=3.95%
lat (msec) : 100=2.39%, 250=0.90%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=1.01%, sys=3.06%, ctx=39314, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=206746/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4015
write: io=826752KB, bw=13779KB/s, iops=3444 , runt= 60003msec
slat (usec): min=4 , max=116274 , avg=108.34, stdev=2274.99
clat (usec): min=610 , max=1268.9K, avg=9177.07, stdev=20181.03
lat (usec): min=622 , max=1268.9K, avg=9285.61, stdev=20895.62
clat percentiles (usec):
| 1.00th=[ 1416], 5.00th=[ 2128], 10.00th=[ 2608], 20.00th=[ 3248],
| 30.00th=[ 3792], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[35072],
| 99.00th=[94720], 99.50th=[144384], 99.90th=[224256]
bw (KB/s) : min= 6374, max=30040, per=4.35%, avg=13772.31, stdev=4065.37
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=3.91%, 4=29.33%, 10=57.75%, 20=1.68%, 50=3.86%
lat (msec) : 100=2.45%, 250=0.84%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.91%, sys=3.16%, ctx=40150, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=206688/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4016
write: io=774808KB, bw=12899KB/s, iops=3224 , runt= 60066msec
slat (usec): min=4 , max=116276 , avg=122.45, stdev=2450.71
clat (usec): min=492 , max=1488.5K, avg=9785.13, stdev=21698.70
lat (usec): min=505 , max=1488.5K, avg=9907.78, stdev=22471.59
clat percentiles (usec):
| 1.00th=[ 1400], 5.00th=[ 2096], 10.00th=[ 2544], 20.00th=[ 3280],
| 30.00th=[ 3888], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9664], 95.00th=[38144],
| 99.00th=[101888], 99.50th=[154624], 99.90th=[228352]
bw (KB/s) : min= 5100, max=24714, per=4.08%, avg=12937.68, stdev=3431.60
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.13%
lat (msec) : 2=4.14%, 4=27.53%, 10=58.41%, 20=1.78%, 50=4.26%
lat (msec) : 100=2.68%, 250=0.97%, 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.95%, sys=2.97%, ctx=39454, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=193702/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4018
write: io=802892KB, bw=13245KB/s, iops=3311 , runt= 60617msec
slat (usec): min=4 , max=115549 , avg=114.45, stdev=2351.84
clat (usec): min=443 , max=1541.4K, avg=9448.96, stdev=20788.49
lat (usec): min=460 , max=1596.1K, avg=9563.61, stdev=21560.78
clat percentiles (usec):
| 1.00th=[ 1240], 5.00th=[ 2008], 10.00th=[ 2544], 20.00th=[ 3248],
| 30.00th=[ 3856], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9152], 95.00th=[36608],
| 99.00th=[98816], 99.50th=[152576], 99.90th=[224256]
bw (KB/s) : min= 1731, max=24256, per=4.20%, avg=13307.98, stdev=3643.41
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.25%
lat (msec) : 2=4.71%, 4=27.47%, 10=58.34%, 20=1.62%, 50=4.11%
lat (msec) : 100=2.48%, 250=0.95%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=3.06%, ctx=40190, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=200723/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4019
write: io=778136KB, bw=12945KB/s, iops=3236 , runt= 60111msec
slat (usec): min=4 , max=135424 , avg=117.66, stdev=2386.05
clat (usec): min=452 , max=1677.3K, avg=9747.29, stdev=22049.42
lat (usec): min=469 , max=1677.4K, avg=9864.91, stdev=22787.60
clat percentiles (usec):
| 1.00th=[ 1384], 5.00th=[ 2128], 10.00th=[ 2576], 20.00th=[ 3248],
| 30.00th=[ 3824], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9408], 95.00th=[37120],
| 99.00th=[113152], 99.50th=[162816], 99.90th=[224256]
bw (KB/s) : min= 6296, max=38856, per=4.10%, avg=12994.39, stdev=4142.39
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.14%
lat (msec) : 2=3.85%, 4=28.97%, 10=57.45%, 20=1.83%, 50=4.00%
lat (msec) : 100=2.63%, 250=1.06%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.85%, sys=3.06%, ctx=40263, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=194534/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=18756MB, aggrb=316840KB/s, minb=13138KB/s, maxb=14976KB/s,
mint=60001msec, maxt=60617msec
Disk stats (read/write):
bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
--
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
