On Wed, Apr 22, 2015 at 4:07 PM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote:
> I am suggesting synthetic workload like fio to run on top of VM to identify where the bottleneck is. For example, if fio is giving decent enough output, I guess ceph layer is doing fine. It is your client that is not driving enough.
After spending a day learning about fio and collecting some data, here
are some results from a new VM running ext4 on a 3TB image backed by
this cluster. The default ZFS block size is 128K and the snapshot
receive operation is basically write-only so the random 128K write
test is probably the closest to our workload and, indeed, performs
very similarly: ~20M/sec throughput and 153 IOPs. (Although really
all the random write tests are pretty bad.) The script used is at the
bottom, and is adapted from one I found on this list.
4K sequential read:
testfile: (groupid=0, jobs=64): err= 0: pid=2893: Thu Apr 23 07:04:13 2015
read : io=83563MB, bw=47538KB/s, iops=11884, runt=1800008msec
clat (usec): min=775, max=617891, avg=5381.74, stdev=3221.32
lat (usec): min=776, max=617892, avg=5382.21, stdev=3221.31
clat percentiles (usec):
| 1.00th=[ 2480], 5.00th=[ 2960], 10.00th=[ 3184], 20.00th=[ 3472],
| 30.00th=[ 3696], 40.00th=[ 3952], 50.00th=[ 4320], 60.00th=[ 5536],
| 70.00th=[ 6432], 80.00th=[ 7072], 90.00th=[ 7968], 95.00th=[ 9664],
| 99.00th=[14400], 99.50th=[17792], 99.90th=[36096], 99.95th=[48384],
| 99.99th=[96768]
bw (KB /s): min= 6, max= 1266, per=1.56%, avg=743.41, stdev=167.14
lat (usec) : 1000=0.01%
lat (msec) : 2=0.15%, 4=41.68%, 10=53.77%, 20=4.05%, 50=0.31%
lat (msec) : 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=0.11%, sys=0.45%, ctx=24873086, majf=0, minf=1843
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=21392078/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=83563MB, aggrb=47537KB/s, minb=47537KB/s, maxb=47537KB/s,
mint=1800008msec, maxt=1800008msec
Disk stats (read/write):
vdb: ios=21391171/11968, merge=0/333, ticks=103378628/812140,
in_queue=104445000, util=100.00%
128K sequential read:
testfile: (groupid=0, jobs=64): err= 0: pid=3065: Thu Apr 23 07:34:14 2015
read : io=393209MB, bw=223681KB/s, iops=1747, runt=1800090msec
clat (msec): min=2, max=5388, avg=36.62, stdev=68.58
lat (msec): min=2, max=5388, avg=36.62, stdev=68.58
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 11], 50.00th=[ 18], 60.00th=[ 28],
| 70.00th=[ 41], 80.00th=[ 59], 90.00th=[ 89], 95.00th=[ 121],
| 99.00th=[ 227], 99.50th=[ 326], 99.90th=[ 840], 99.95th=[ 1172],
| 99.99th=[ 2073]
bw (KB /s): min= 24, max=36207, per=1.65%, avg=3690.81, stdev=3784.08
lat (msec) : 4=24.76%, 10=14.89%, 20=12.70%, 50=23.77%, 100=16.04%
lat (msec) : 250=6.99%, 500=0.58%, 750=0.13%, 1000=0.05%, 2000=0.06%
lat (msec) : >=2000=0.01%
cpu : usr=0.03%, sys=0.14%, ctx=3829128, majf=0, minf=3910
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=3145673/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=393209MB, aggrb=223681KB/s, minb=223681KB/s,
maxb=223681KB/s, mint=1800090msec, maxt=1800090msec
Disk stats (read/write):
vdb: ios=3145617/6096, merge=0/170, ticks=95140524/632704,
in_queue=95776824, util=100.00%
8M sequential read:
testfile: (groupid=0, jobs=64): err= 0: pid=3239: Thu Apr 23 08:04:16 2015
read : io=393920MB, bw=223916KB/s, iops=27, runt=1801449msec
clat (msec): min=65, max=23189, avg=2340.68, stdev=1351.01
lat (msec): min=65, max=23189, avg=2340.68, stdev=1351.01
clat percentiles (msec):
| 1.00th=[ 204], 5.00th=[ 474], 10.00th=[ 791], 20.00th=[ 1221],
| 30.00th=[ 1532], 40.00th=[ 1844], 50.00th=[ 2147], 60.00th=[ 2474],
| 70.00th=[ 2868], 80.00th=[ 3359], 90.00th=[ 4080], 95.00th=[ 4817],
| 99.00th=[ 6456], 99.50th=[ 7177], 99.90th=[ 9110], 99.95th=[10159],
| 99.99th=[12518]
bw (KB /s): min= 353, max=52012, per=2.09%, avg=4674.81, stdev=3150.53
lat (msec) : 100=0.09%, 250=1.56%, 500=3.76%, 750=3.90%, 1000=4.99%
lat (msec) : 2000=30.78%, >=2000=54.92%
cpu : usr=0.00%, sys=0.23%, ctx=41395542, majf=0, minf=34821
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=49240/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=393920MB, aggrb=223916KB/s, minb=223916KB/s,
maxb=223916KB/s, mint=1801449msec, maxt=1801449msec
Disk stats (read/write):
vdb: ios=837073/0, merge=0/0, ticks=113067700/0, in_queue=113069512,
util=100.00%
4k sequential write:
testfile: (groupid=0, jobs=64): err= 0: pid=3414: Thu Apr 23 08:34:16 2015
write: io=7898.4MB, bw=4493.2KB/s, iops=1123, runt=1800051msec
clat (msec): min=2, max=1524, avg=56.97, stdev=65.08
lat (msec): min=2, max=1524, avg=56.97, stdev=65.08
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 9], 10.00th=[ 14], 20.00th=[ 17],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 25], 60.00th=[ 43],
| 70.00th=[ 64], 80.00th=[ 93], 90.00th=[ 147], 95.00th=[ 194],
| 99.00th=[ 289], 99.50th=[ 334], 99.90th=[ 445], 99.95th=[ 506],
| 99.99th=[ 816]
bw (KB /s): min= 2, max= 439, per=1.60%, avg=71.94, stdev=54.95
lat (msec) : 4=2.26%, 10=4.21%, 20=38.51%, 50=18.38%, 100=18.16%
lat (msec) : 250=16.55%, 500=1.88%, 750=0.04%, 1000=0.01%, 2000=0.01%
cpu : usr=0.02%, sys=0.07%, ctx=2562899, majf=0, minf=1897
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=2021970/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=7898.4MB, aggrb=4493KB/s, minb=4493KB/s, maxb=4493KB/s,
mint=1800051msec, maxt=1800051msec
Disk stats (read/write):
vdb: ios=0/2022593, merge=0/1795, ticks=0/103422168,
in_queue=103434508, util=100.00%
128K sequential write:
testfile: (groupid=0, jobs=64): err= 0: pid=3587: Thu Apr 23 09:04:17 2015
write: io=127184MB, bw=72347KB/s, iops=565, runt=1800179msec
clat (msec): min=4, max=2038, avg=113.21, stdev=133.77
lat (msec): min=4, max=2038, avg=113.22, stdev=133.77
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 13], 20.00th=[ 19],
| 30.00th=[ 21], 40.00th=[ 36], 50.00th=[ 57], 60.00th=[ 83],
| 70.00th=[ 137], 80.00th=[ 208], 90.00th=[ 297], 95.00th=[ 379],
| 99.00th=[ 586], 99.50th=[ 668], 99.90th=[ 922], 99.95th=[ 1045],
| 99.99th=[ 1401]
bw (KB /s): min= 64, max=11086, per=1.68%, avg=1214.97, stdev=1228.99
lat (msec) : 10=7.19%, 20=21.62%, 50=17.98%, 100=17.32%, 250=21.29%
lat (msec) : 500=12.68%, 750=1.65%, 1000=0.22%, 2000=0.06%, >=2000=0.01%
cpu : usr=0.02%, sys=0.05%, ctx=1271554, majf=0, minf=1828
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=1017474/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=127184MB, aggrb=72346KB/s, minb=72346KB/s, maxb=72346KB/s,
mint=1800179msec, maxt=1800179msec
Disk stats (read/write):
vdb: ios=0/1018181, merge=0/1800, ticks=0/103424596,
in_queue=103433068, util=100.00%
8M sequential write:
testfile: (groupid=0, jobs=64): err= 0: pid=3756: Thu Apr 23 09:34:20 2015
write: io=225824MB, bw=128258KB/s, iops=15, runt=1802960msec
clat (msec): min=114, max=28746, avg=4084.27, stdev=3047.60
lat (msec): min=114, max=28747, avg=4084.77, stdev=3047.61
clat percentiles (msec):
| 1.00th=[ 247], 5.00th=[ 545], 10.00th=[ 914], 20.00th=[ 1549],
| 30.00th=[ 2147], 40.00th=[ 2737], 50.00th=[ 3392], 60.00th=[ 4113],
| 70.00th=[ 5014], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[10028],
| 99.00th=[14222], 99.50th=[16188], 99.90th=[16712], 99.95th=[16712],
| 99.99th=[16712]
bw (KB /s): min= 284, max=32572, per=2.67%, avg=3422.87, stdev=3126.28
lat (msec) : 250=1.06%, 500=3.44%, 750=3.27%, 1000=3.66%, 2000=16.12%
lat (msec) : >=2000=72.46%
cpu : usr=0.01%, sys=0.04%, ctx=4460079, majf=0, minf=1809
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=28228/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=225824MB, aggrb=128257KB/s, minb=128257KB/s,
maxb=128257KB/s, mint=1802960msec, maxt=1802960msec
Disk stats (read/write):
vdb: ios=0/480528, merge=0/1770, ticks=0/113458632,
in_queue=113471896, util=100.00%
4K random read:
testfile: (groupid=0, jobs=64): err= 0: pid=3932: Thu Apr 23 10:04:21 2015
read : io=10473MB, bw=5957.6KB/s, iops=1489, runt=1800151msec
clat (usec): min=861, max=21197K, avg=42961.55, stdev=102880.56
lat (usec): min=861, max=21197K, avg=42962.08, stdev=102880.57
clat percentiles (usec):
| 1.00th=[ 1496], 5.00th=[ 8768], 10.00th=[11072], 20.00th=[14016],
| 30.00th=[16768], 40.00th=[20096], 50.00th=[24192], 60.00th=[30080],
| 70.00th=[38144], 80.00th=[51968], 90.00th=[81408], 95.00th=[121344],
| 99.00th=[292864], 99.50th=[444416], 99.90th=[1122304], 99.95th=[1646592],
| 99.99th=[3686400]
bw (KB /s): min= 0, max= 269, per=1.68%, avg=99.91, stdev=43.00
lat (usec) : 1000=0.01%
lat (msec) : 2=1.41%, 4=0.20%, 10=5.71%, 20=32.50%, 50=39.00%
lat (msec) : 100=14.05%, 250=5.81%, 500=0.91%, 750=0.20%, 1000=0.08%
lat (msec) : 2000=0.09%, >=2000=0.03%
cpu : usr=0.03%, sys=0.10%, ctx=3044301, majf=0, minf=1985
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=2681119/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=10473MB, aggrb=5957KB/s, minb=5957KB/s, maxb=5957KB/s,
mint=1800151msec, maxt=1800151msec
Disk stats (read/write):
vdb: ios=2681117/3, merge=0/9, ticks=96400660/812,
in_queue=96405480, util=100.00%
128K random read:
testfile: (groupid=0, jobs=64): err= 0: pid=4100: Thu Apr 23 10:34:22 2015
read : io=216528MB, bw=123166KB/s, iops=962, runt=1800215msec
clat (msec): min=2, max=14946, avg=66.50, stdev=168.09
lat (msec): min=2, max=14946, avg=66.50, stdev=168.09
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 21],
| 30.00th=[ 24], 40.00th=[ 28], 50.00th=[ 33], 60.00th=[ 41],
| 70.00th=[ 52], 80.00th=[ 73], 90.00th=[ 125], 95.00th=[ 206],
| 99.00th=[ 537], 99.50th=[ 766], 99.90th=[ 2147], 99.95th=[ 3130],
| 99.99th=[ 5538]
bw (KB /s): min= 8, max= 5841, per=1.77%, avg=2183.13, stdev=1042.43
lat (msec) : 4=0.51%, 10=0.96%, 20=18.44%, 50=48.63%, 100=18.20%
lat (msec) : 250=9.47%, 500=2.65%, 750=0.62%, 1000=0.19%, 2000=0.22%
lat (msec) : >=2000=0.11%
cpu : usr=0.02%, sys=0.09%, ctx=1934535, majf=0, minf=3970
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=1732226/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=216528MB, aggrb=123165KB/s, minb=123165KB/s,
maxb=123165KB/s, mint=1800215msec, maxt=1800215msec
Disk stats (read/write):
vdb: ios=1732223/0, merge=0/0, ticks=94884888/0, in_queue=94885996,
util=100.00%
testfile: (groupid=0, jobs=64): err= 0: pid=4271: Thu Apr 23 11:04:24 2015
read : io=391040MB, bw=222304KB/s, iops=27, runt=1801253msec
clat (msec): min=74, max=19112, avg=2357.89, stdev=1296.63
lat (msec): min=74, max=19112, avg=2357.90, stdev=1296.63
clat percentiles (msec):
| 1.00th=[ 223], 5.00th=[ 553], 10.00th=[ 889], 20.00th=[ 1287],
| 30.00th=[ 1598], 40.00th=[ 1893], 50.00th=[ 2180], 60.00th=[ 2507],
| 70.00th=[ 2835], 80.00th=[ 3294], 90.00th=[ 4015], 95.00th=[ 4686],
| 99.00th=[ 6325], 99.50th=[ 7046], 99.90th=[ 9241], 99.95th=[10159],
| 99.99th=[12649]
bw (KB /s): min= 428, max=47261, per=2.04%, avg=4540.89, stdev=2927.76
lat (msec) : 100=0.11%, 250=1.17%, 500=3.14%, 750=3.28%, 1000=4.58%
lat (msec) : 2000=31.36%, >=2000=56.35%
cpu : usr=0.00%, sys=0.23%, ctx=41775294, majf=0, minf=34876
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=48880/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=391040MB, aggrb=222303KB/s, minb=222303KB/s,
maxb=222303KB/s, mint=1801253msec, maxt=1801253msec
Disk stats (read/write):
vdb: ios=830917/0, merge=0/0, ticks=113352364/0, in_queue=113356088,
util=100.00%
4K random write:
testfile: (groupid=0, jobs=64): err= 0: pid=4446: Thu Apr 23 11:34:25 2015
write: io=1078.5MB, bw=627868B/s, iops=153, runt=1801129msec
clat (msec): min=3, max=5536, avg=417.30, stdev=374.78
lat (msec): min=3, max=5536, avg=417.30, stdev=374.78
clat percentiles (msec):
| 1.00th=[ 10], 5.00th=[ 34], 10.00th=[ 46], 20.00th=[ 82],
| 30.00th=[ 139], 40.00th=[ 239], 50.00th=[ 347], 60.00th=[ 449],
| 70.00th=[ 553], 80.00th=[ 676], 90.00th=[ 881], 95.00th=[ 1106],
| 99.00th=[ 1680], 99.50th=[ 1991], 99.90th=[ 2737], 99.95th=[ 3032],
| 99.99th=[ 3884]
bw (KB /s): min= 0, max= 134, per=1.70%, avg=10.44, stdev= 9.57
lat (msec) : 4=0.05%, 10=0.97%, 20=0.75%, 50=9.73%, 100=12.33%
lat (msec) : 250=17.23%, 500=24.05%, 750=19.74%, 1000=8.22%, 2000=6.45%
lat (msec) : >=2000=0.48%
cpu : usr=0.00%, sys=0.01%, ctx=393118, majf=0, minf=1872
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=276092/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=1078.5MB, aggrb=613KB/s, minb=613KB/s, maxb=613KB/s,
mint=1801129msec, maxt=1801129msec
Disk stats (read/write):
vdb: ios=0/276802, merge=0/1790, ticks=0/102253144,
in_queue=102257136, util=100.00%
128K random write:
testfile: (groupid=0, jobs=64): err= 0: pid=4617: Thu Apr 23 12:04:26 2015
write: io=34500MB, bw=19621KB/s, iops=153, runt=1800496msec
clat (msec): min=4, max=5320, avg=417.44, stdev=370.55
lat (msec): min=5, max=5320, avg=417.45, stdev=370.55
clat percentiles (msec):
| 1.00th=[ 8], 5.00th=[ 31], 10.00th=[ 41], 20.00th=[ 71],
| 30.00th=[ 129], 40.00th=[ 241], 50.00th=[ 359], 60.00th=[ 461],
| 70.00th=[ 562], 80.00th=[ 685], 90.00th=[ 889], 95.00th=[ 1106],
| 99.00th=[ 1598], 99.50th=[ 1876], 99.90th=[ 2474], 99.95th=[ 2769],
| 99.99th=[ 3130]
bw (KB /s): min= 24, max= 4942, per=1.77%, avg=348.17, stdev=324.99
lat (msec) : 10=1.33%, 20=1.27%, 50=11.42%, 100=11.97%, 250=14.88%
lat (msec) : 500=23.01%, 750=20.36%, 1000=8.72%, 2000=6.69%, >=2000=0.35%
cpu : usr=0.01%, sys=0.01%, ctx=387874, majf=0, minf=1886
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=276001/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=34500MB, aggrb=19621KB/s, minb=19621KB/s, maxb=19621KB/s,
mint=1800496msec, maxt=1800496msec
Disk stats (read/write):
vdb: ios=0/276686, merge=0/1795, ticks=0/102181152,
in_queue=102206116, util=100.00%
8M random write:
testfile: (groupid=0, jobs=64): err= 0: pid=4783: Thu Apr 23 12:34:30 2015
write: io=224912MB, bw=127712KB/s, iops=15, runt=1803356msec
clat (msec): min=112, max=26723, avg=4100.95, stdev=3006.58
lat (msec): min=112, max=26724, avg=4101.45, stdev=3006.59
clat percentiles (msec):
| 1.00th=[ 241], 5.00th=[ 570], 10.00th=[ 914], 20.00th=[ 1582],
| 30.00th=[ 2180], 40.00th=[ 2802], 50.00th=[ 3425], 60.00th=[ 4178],
| 70.00th=[ 5080], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[ 9896],
| 99.00th=[13829], 99.50th=[15795], 99.90th=[16712], 99.95th=[16712],
| 99.99th=[16712]
bw (KB /s): min= 306, max=32125, per=2.66%, avg=3403.48, stdev=3078.10
lat (msec) : 250=1.08%, 500=3.00%, 750=3.42%, 1000=3.78%, 2000=15.69%
lat (msec) : >=2000=73.03%
cpu : usr=0.01%, sys=0.04%, ctx=4404542, majf=0, minf=1863
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=28114/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=224912MB, aggrb=127711KB/s, minb=127711KB/s,
maxb=127711KB/s, mint=1803356msec, maxt=1803356msec
Disk stats (read/write):
vdb: ios=0/478659, merge=0/1766, ticks=0/113423852,
in_queue=113426920, util=100.00%
The test script:
#!/bin/sh
Directory=/mnt
RunTime=1800
Size=30G
NumJobs=64
for Action in read write randread randwrite
do
for BlockSize in 4k 128k 8m
do
OutputFile=fio-${Action}-${BlockSize}-${RunTime}-${Size}-${NumJobs}.txt
fio --directory=$Directory --direct=1 \
--rw=$Action --bs=$BlockSize --size=$Size --numjobs=$NumJobs \
--runtime=$RunTime --group_reporting --name=testfile \
--output=$OutputFile
done
done
Thanks!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
