David, With the similar 128K profile I am getting ~200MB/s bandwidth with entire OSD on SSD..I never tested with HDDs, but, it seems you are reaching Ceph's limit on this. Probably, nothing wrong in your setup ! Thanks & Regards Somnath -----Original Message----- From: jdavidlists@xxxxxxxxx [mailto:jdavidlists@xxxxxxxxx] On Behalf Of J David Sent: Thursday, April 23, 2015 9:56 AM To: Somnath Roy Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: Having trouble getting good performance On Wed, Apr 22, 2015 at 4:07 PM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: > I am suggesting synthetic workload like fio to run on top of VM to identify where the bottleneck is. For example, if fio is giving decent enough output, I guess ceph layer is doing fine. It is your client that is not driving enough. After spending a day learning about fio and collecting some data, here are some results from a new VM running ext4 on a 3TB image backed by this cluster. The default ZFS block size is 128K and the snapshot receive operation is basically write-only so the random 128K write test is probably the closest to our workload and, indeed, performs very similarly: ~20M/sec throughput and 153 IOPs. (Although really all the random write tests are pretty bad.) The script used is at the bottom, and is adapted from one I found on this list. 4K sequential read: testfile: (groupid=0, jobs=64): err= 0: pid=2893: Thu Apr 23 07:04:13 2015 read : io=83563MB, bw=47538KB/s, iops=11884, runt=1800008msec clat (usec): min=775, max=617891, avg=5381.74, stdev=3221.32 lat (usec): min=776, max=617892, avg=5382.21, stdev=3221.31 clat percentiles (usec): | 1.00th=[ 2480], 5.00th=[ 2960], 10.00th=[ 3184], 20.00th=[ 3472], | 30.00th=[ 3696], 40.00th=[ 3952], 50.00th=[ 4320], 60.00th=[ 5536], | 70.00th=[ 6432], 80.00th=[ 7072], 90.00th=[ 7968], 95.00th=[ 9664], | 99.00th=[14400], 99.50th=[17792], 99.90th=[36096], 99.95th=[48384], | 99.99th=[96768] bw (KB /s): min= 6, max= 1266, per=1.56%, avg=743.41, stdev=167.14 lat (usec) : 1000=0.01% lat (msec) : 2=0.15%, 4=41.68%, 10=53.77%, 20=4.05%, 50=0.31% lat (msec) : 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=0.11%, sys=0.45%, ctx=24873086, majf=0, minf=1843 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=21392078/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=83563MB, aggrb=47537KB/s, minb=47537KB/s, maxb=47537KB/s, mint=1800008msec, maxt=1800008msec Disk stats (read/write): vdb: ios=21391171/11968, merge=0/333, ticks=103378628/812140, in_queue=104445000, util=100.00% 128K sequential read: testfile: (groupid=0, jobs=64): err= 0: pid=3065: Thu Apr 23 07:34:14 2015 read : io=393209MB, bw=223681KB/s, iops=1747, runt=1800090msec clat (msec): min=2, max=5388, avg=36.62, stdev=68.58 lat (msec): min=2, max=5388, avg=36.62, stdev=68.58 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 11], 50.00th=[ 18], 60.00th=[ 28], | 70.00th=[ 41], 80.00th=[ 59], 90.00th=[ 89], 95.00th=[ 121], | 99.00th=[ 227], 99.50th=[ 326], 99.90th=[ 840], 99.95th=[ 1172], | 99.99th=[ 2073] bw (KB /s): min= 24, max=36207, per=1.65%, avg=3690.81, stdev=3784.08 lat (msec) : 4=24.76%, 10=14.89%, 20=12.70%, 50=23.77%, 100=16.04% lat (msec) : 250=6.99%, 500=0.58%, 750=0.13%, 1000=0.05%, 2000=0.06% lat (msec) : >=2000=0.01% cpu : usr=0.03%, sys=0.14%, ctx=3829128, majf=0, minf=3910 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=3145673/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=393209MB, aggrb=223681KB/s, minb=223681KB/s, maxb=223681KB/s, mint=1800090msec, maxt=1800090msec Disk stats (read/write): vdb: ios=3145617/6096, merge=0/170, ticks=95140524/632704, in_queue=95776824, util=100.00% 8M sequential read: testfile: (groupid=0, jobs=64): err= 0: pid=3239: Thu Apr 23 08:04:16 2015 read : io=393920MB, bw=223916KB/s, iops=27, runt=1801449msec clat (msec): min=65, max=23189, avg=2340.68, stdev=1351.01 lat (msec): min=65, max=23189, avg=2340.68, stdev=1351.01 clat percentiles (msec): | 1.00th=[ 204], 5.00th=[ 474], 10.00th=[ 791], 20.00th=[ 1221], | 30.00th=[ 1532], 40.00th=[ 1844], 50.00th=[ 2147], 60.00th=[ 2474], | 70.00th=[ 2868], 80.00th=[ 3359], 90.00th=[ 4080], 95.00th=[ 4817], | 99.00th=[ 6456], 99.50th=[ 7177], 99.90th=[ 9110], 99.95th=[10159], | 99.99th=[12518] bw (KB /s): min= 353, max=52012, per=2.09%, avg=4674.81, stdev=3150.53 lat (msec) : 100=0.09%, 250=1.56%, 500=3.76%, 750=3.90%, 1000=4.99% lat (msec) : 2000=30.78%, >=2000=54.92% cpu : usr=0.00%, sys=0.23%, ctx=41395542, majf=0, minf=34821 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=49240/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=393920MB, aggrb=223916KB/s, minb=223916KB/s, maxb=223916KB/s, mint=1801449msec, maxt=1801449msec Disk stats (read/write): vdb: ios=837073/0, merge=0/0, ticks=113067700/0, in_queue=113069512, util=100.00% 4k sequential write: testfile: (groupid=0, jobs=64): err= 0: pid=3414: Thu Apr 23 08:34:16 2015 write: io=7898.4MB, bw=4493.2KB/s, iops=1123, runt=1800051msec clat (msec): min=2, max=1524, avg=56.97, stdev=65.08 lat (msec): min=2, max=1524, avg=56.97, stdev=65.08 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 9], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 25], 60.00th=[ 43], | 70.00th=[ 64], 80.00th=[ 93], 90.00th=[ 147], 95.00th=[ 194], | 99.00th=[ 289], 99.50th=[ 334], 99.90th=[ 445], 99.95th=[ 506], | 99.99th=[ 816] bw (KB /s): min= 2, max= 439, per=1.60%, avg=71.94, stdev=54.95 lat (msec) : 4=2.26%, 10=4.21%, 20=38.51%, 50=18.38%, 100=18.16% lat (msec) : 250=16.55%, 500=1.88%, 750=0.04%, 1000=0.01%, 2000=0.01% cpu : usr=0.02%, sys=0.07%, ctx=2562899, majf=0, minf=1897 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=2021970/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=7898.4MB, aggrb=4493KB/s, minb=4493KB/s, maxb=4493KB/s, mint=1800051msec, maxt=1800051msec Disk stats (read/write): vdb: ios=0/2022593, merge=0/1795, ticks=0/103422168, in_queue=103434508, util=100.00% 128K sequential write: testfile: (groupid=0, jobs=64): err= 0: pid=3587: Thu Apr 23 09:04:17 2015 write: io=127184MB, bw=72347KB/s, iops=565, runt=1800179msec clat (msec): min=4, max=2038, avg=113.21, stdev=133.77 lat (msec): min=4, max=2038, avg=113.22, stdev=133.77 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 13], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 36], 50.00th=[ 57], 60.00th=[ 83], | 70.00th=[ 137], 80.00th=[ 208], 90.00th=[ 297], 95.00th=[ 379], | 99.00th=[ 586], 99.50th=[ 668], 99.90th=[ 922], 99.95th=[ 1045], | 99.99th=[ 1401] bw (KB /s): min= 64, max=11086, per=1.68%, avg=1214.97, stdev=1228.99 lat (msec) : 10=7.19%, 20=21.62%, 50=17.98%, 100=17.32%, 250=21.29% lat (msec) : 500=12.68%, 750=1.65%, 1000=0.22%, 2000=0.06%, >=2000=0.01% cpu : usr=0.02%, sys=0.05%, ctx=1271554, majf=0, minf=1828 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=1017474/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=127184MB, aggrb=72346KB/s, minb=72346KB/s, maxb=72346KB/s, mint=1800179msec, maxt=1800179msec Disk stats (read/write): vdb: ios=0/1018181, merge=0/1800, ticks=0/103424596, in_queue=103433068, util=100.00% 8M sequential write: testfile: (groupid=0, jobs=64): err= 0: pid=3756: Thu Apr 23 09:34:20 2015 write: io=225824MB, bw=128258KB/s, iops=15, runt=1802960msec clat (msec): min=114, max=28746, avg=4084.27, stdev=3047.60 lat (msec): min=114, max=28747, avg=4084.77, stdev=3047.61 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 545], 10.00th=[ 914], 20.00th=[ 1549], | 30.00th=[ 2147], 40.00th=[ 2737], 50.00th=[ 3392], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[10028], | 99.00th=[14222], 99.50th=[16188], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw (KB /s): min= 284, max=32572, per=2.67%, avg=3422.87, stdev=3126.28 lat (msec) : 250=1.06%, 500=3.44%, 750=3.27%, 1000=3.66%, 2000=16.12% lat (msec) : >=2000=72.46% cpu : usr=0.01%, sys=0.04%, ctx=4460079, majf=0, minf=1809 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=28228/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=225824MB, aggrb=128257KB/s, minb=128257KB/s, maxb=128257KB/s, mint=1802960msec, maxt=1802960msec Disk stats (read/write): vdb: ios=0/480528, merge=0/1770, ticks=0/113458632, in_queue=113471896, util=100.00% 4K random read: testfile: (groupid=0, jobs=64): err= 0: pid=3932: Thu Apr 23 10:04:21 2015 read : io=10473MB, bw=5957.6KB/s, iops=1489, runt=1800151msec clat (usec): min=861, max=21197K, avg=42961.55, stdev=102880.56 lat (usec): min=861, max=21197K, avg=42962.08, stdev=102880.57 clat percentiles (usec): | 1.00th=[ 1496], 5.00th=[ 8768], 10.00th=[11072], 20.00th=[14016], | 30.00th=[16768], 40.00th=[20096], 50.00th=[24192], 60.00th=[30080], | 70.00th=[38144], 80.00th=[51968], 90.00th=[81408], 95.00th=[121344], | 99.00th=[292864], 99.50th=[444416], 99.90th=[1122304], 99.95th=[1646592], | 99.99th=[3686400] bw (KB /s): min= 0, max= 269, per=1.68%, avg=99.91, stdev=43.00 lat (usec) : 1000=0.01% lat (msec) : 2=1.41%, 4=0.20%, 10=5.71%, 20=32.50%, 50=39.00% lat (msec) : 100=14.05%, 250=5.81%, 500=0.91%, 750=0.20%, 1000=0.08% lat (msec) : 2000=0.09%, >=2000=0.03% cpu : usr=0.03%, sys=0.10%, ctx=3044301, majf=0, minf=1985 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=2681119/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=10473MB, aggrb=5957KB/s, minb=5957KB/s, maxb=5957KB/s, mint=1800151msec, maxt=1800151msec Disk stats (read/write): vdb: ios=2681117/3, merge=0/9, ticks=96400660/812, in_queue=96405480, util=100.00% 128K random read: testfile: (groupid=0, jobs=64): err= 0: pid=4100: Thu Apr 23 10:34:22 2015 read : io=216528MB, bw=123166KB/s, iops=962, runt=1800215msec clat (msec): min=2, max=14946, avg=66.50, stdev=168.09 lat (msec): min=2, max=14946, avg=66.50, stdev=168.09 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 28], 50.00th=[ 33], 60.00th=[ 41], | 70.00th=[ 52], 80.00th=[ 73], 90.00th=[ 125], 95.00th=[ 206], | 99.00th=[ 537], 99.50th=[ 766], 99.90th=[ 2147], 99.95th=[ 3130], | 99.99th=[ 5538] bw (KB /s): min= 8, max= 5841, per=1.77%, avg=2183.13, stdev=1042.43 lat (msec) : 4=0.51%, 10=0.96%, 20=18.44%, 50=48.63%, 100=18.20% lat (msec) : 250=9.47%, 500=2.65%, 750=0.62%, 1000=0.19%, 2000=0.22% lat (msec) : >=2000=0.11% cpu : usr=0.02%, sys=0.09%, ctx=1934535, majf=0, minf=3970 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1732226/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=216528MB, aggrb=123165KB/s, minb=123165KB/s, maxb=123165KB/s, mint=1800215msec, maxt=1800215msec Disk stats (read/write): vdb: ios=1732223/0, merge=0/0, ticks=94884888/0, in_queue=94885996, util=100.00% testfile: (groupid=0, jobs=64): err= 0: pid=4271: Thu Apr 23 11:04:24 2015 read : io=391040MB, bw=222304KB/s, iops=27, runt=1801253msec clat (msec): min=74, max=19112, avg=2357.89, stdev=1296.63 lat (msec): min=74, max=19112, avg=2357.90, stdev=1296.63 clat percentiles (msec): | 1.00th=[ 223], 5.00th=[ 553], 10.00th=[ 889], 20.00th=[ 1287], | 30.00th=[ 1598], 40.00th=[ 1893], 50.00th=[ 2180], 60.00th=[ 2507], | 70.00th=[ 2835], 80.00th=[ 3294], 90.00th=[ 4015], 95.00th=[ 4686], | 99.00th=[ 6325], 99.50th=[ 7046], 99.90th=[ 9241], 99.95th=[10159], | 99.99th=[12649] bw (KB /s): min= 428, max=47261, per=2.04%, avg=4540.89, stdev=2927.76 lat (msec) : 100=0.11%, 250=1.17%, 500=3.14%, 750=3.28%, 1000=4.58% lat (msec) : 2000=31.36%, >=2000=56.35% cpu : usr=0.00%, sys=0.23%, ctx=41775294, majf=0, minf=34876 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=48880/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=391040MB, aggrb=222303KB/s, minb=222303KB/s, maxb=222303KB/s, mint=1801253msec, maxt=1801253msec Disk stats (read/write): vdb: ios=830917/0, merge=0/0, ticks=113352364/0, in_queue=113356088, util=100.00% 4K random write: testfile: (groupid=0, jobs=64): err= 0: pid=4446: Thu Apr 23 11:34:25 2015 write: io=1078.5MB, bw=627868B/s, iops=153, runt=1801129msec clat (msec): min=3, max=5536, avg=417.30, stdev=374.78 lat (msec): min=3, max=5536, avg=417.30, stdev=374.78 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 34], 10.00th=[ 46], 20.00th=[ 82], | 30.00th=[ 139], 40.00th=[ 239], 50.00th=[ 347], 60.00th=[ 449], | 70.00th=[ 553], 80.00th=[ 676], 90.00th=[ 881], 95.00th=[ 1106], | 99.00th=[ 1680], 99.50th=[ 1991], 99.90th=[ 2737], 99.95th=[ 3032], | 99.99th=[ 3884] bw (KB /s): min= 0, max= 134, per=1.70%, avg=10.44, stdev= 9.57 lat (msec) : 4=0.05%, 10=0.97%, 20=0.75%, 50=9.73%, 100=12.33% lat (msec) : 250=17.23%, 500=24.05%, 750=19.74%, 1000=8.22%, 2000=6.45% lat (msec) : >=2000=0.48% cpu : usr=0.00%, sys=0.01%, ctx=393118, majf=0, minf=1872 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=276092/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=1078.5MB, aggrb=613KB/s, minb=613KB/s, maxb=613KB/s, mint=1801129msec, maxt=1801129msec Disk stats (read/write): vdb: ios=0/276802, merge=0/1790, ticks=0/102253144, in_queue=102257136, util=100.00% 128K random write: testfile: (groupid=0, jobs=64): err= 0: pid=4617: Thu Apr 23 12:04:26 2015 write: io=34500MB, bw=19621KB/s, iops=153, runt=1800496msec clat (msec): min=4, max=5320, avg=417.44, stdev=370.55 lat (msec): min=5, max=5320, avg=417.45, stdev=370.55 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 31], 10.00th=[ 41], 20.00th=[ 71], | 30.00th=[ 129], 40.00th=[ 241], 50.00th=[ 359], 60.00th=[ 461], | 70.00th=[ 562], 80.00th=[ 685], 90.00th=[ 889], 95.00th=[ 1106], | 99.00th=[ 1598], 99.50th=[ 1876], 99.90th=[ 2474], 99.95th=[ 2769], | 99.99th=[ 3130] bw (KB /s): min= 24, max= 4942, per=1.77%, avg=348.17, stdev=324.99 lat (msec) : 10=1.33%, 20=1.27%, 50=11.42%, 100=11.97%, 250=14.88% lat (msec) : 500=23.01%, 750=20.36%, 1000=8.72%, 2000=6.69%, >=2000=0.35% cpu : usr=0.01%, sys=0.01%, ctx=387874, majf=0, minf=1886 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=276001/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=34500MB, aggrb=19621KB/s, minb=19621KB/s, maxb=19621KB/s, mint=1800496msec, maxt=1800496msec Disk stats (read/write): vdb: ios=0/276686, merge=0/1795, ticks=0/102181152, in_queue=102206116, util=100.00% 8M random write: testfile: (groupid=0, jobs=64): err= 0: pid=4783: Thu Apr 23 12:34:30 2015 write: io=224912MB, bw=127712KB/s, iops=15, runt=1803356msec clat (msec): min=112, max=26723, avg=4100.95, stdev=3006.58 lat (msec): min=112, max=26724, avg=4101.45, stdev=3006.59 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 570], 10.00th=[ 914], 20.00th=[ 1582], | 30.00th=[ 2180], 40.00th=[ 2802], 50.00th=[ 3425], 60.00th=[ 4178], | 70.00th=[ 5080], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[13829], 99.50th=[15795], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw (KB /s): min= 306, max=32125, per=2.66%, avg=3403.48, stdev=3078.10 lat (msec) : 250=1.08%, 500=3.00%, 750=3.42%, 1000=3.78%, 2000=15.69% lat (msec) : >=2000=73.03% cpu : usr=0.01%, sys=0.04%, ctx=4404542, majf=0, minf=1863 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=28114/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=224912MB, aggrb=127711KB/s, minb=127711KB/s, maxb=127711KB/s, mint=1803356msec, maxt=1803356msec Disk stats (read/write): vdb: ios=0/478659, merge=0/1766, ticks=0/113423852, in_queue=113426920, util=100.00% The test script: #!/bin/sh Directory=/mnt RunTime=1800 Size=30G NumJobs=64 for Action in read write randread randwrite do for BlockSize in 4k 128k 8m do OutputFile=fio-${Action}-${BlockSize}-${RunTime}-${Size}-${NumJobs}.txt fio --directory=$Directory --direct=1 \ --rw=$Action --bs=$BlockSize --size=$Size --numjobs=$NumJobs \ --runtime=$RunTime --group_reporting --name=testfile \ --output=$OutputFile done done Thanks! ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |