|
Hi Gilles, Thanks a lot for the answer, i've made a new benchmark test with bio, i've used the following configuration for the test: ; Four threads, two query, two writers. [global] rw=randread size=256m directory=/mnt/ceph-btrfs-test ioengine=libaio iodepth=4 invalidate=1 direct=1 [bgwriter] rw=randwrite iodepth=32 [queryA] iodepth=1 ioengine=mmap direct=0 thinktime=3 [queryB] iodepth=1 ioengine=mmap direct=0 thinktime=5 [bgupdater] rw=randrw iodepth=16 thinktime=40 size=32m The results: CEPH Cluster: root@ceph-deploy01:/home/ceph/ceph-cluster# fio four-threads-randio.fio bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32 queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1 queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1 bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16 2.0.8 Starting 4 processes bgwriter: Laying out IO file(s) (1 file(s) / 256MB) queryA: Laying out IO file(s) (1 file(s) / 256MB) queryB: Laying out IO file(s) (1 file(s) / 256MB) bgupdater: Laying out IO file(s) (1 file(s) / 32MB) Jobs: 2 (f=2): [_rr_] [99.8% done] [1164K/0K /s] [291 /0 iops] [eta 00m:01s] bgwriter: (groupid=0, jobs=1): err= 0: pid=10481 write: io=262144KB, bw=3990.6KB/s, iops=997 , runt= 65691msec slat (usec): min=35 , max=35877 , avg=131.07, stdev=284.30 clat (msec): min=3 , max=81 , avg=31.94, stdev= 6.72 lat (msec): min=3 , max=82 , avg=32.07, stdev= 6.70 clat percentiles (usec): | 1.00th=[11456], 5.00th=[22400], 10.00th=[25472], 20.00th=[27520], | 30.00th=[29056], 40.00th=[30080], 50.00th=[31104], 60.00th=[32384], | 70.00th=[34048], 80.00th=[36608], 90.00th=[41216], 95.00th=[43776], | 99.00th=[48896], 99.50th=[51968], 99.90th=[59136], 99.95th=[62208], | 99.99th=[74240] bw (KB/s) : min= 2909, max= 4525, per=94.09%, avg=3992.26, stdev=481.59 lat (msec) : 4=0.01%, 10=0.78%, 20=2.23%, 50=96.20%, 100=0.78% cpu : usr=0.85%, sys=11.90%, ctx=116709, majf=0, minf=20 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0 queryA: (groupid=0, jobs=1): err= 0: pid=10482 read : io=262144KB, bw=517027 B/s, iops=126 , runt=519190msec clat (usec): min=700 , max=46513 , avg=7845.19, stdev=13054.61 lat (usec): min=701 , max=46514 , avg=7845.79, stdev=13054.61 clat percentiles (usec): | 1.00th=[ 1208], 5.00th=[ 1400], 10.00th=[ 1448], 20.00th=[ 1544], | 30.00th=[ 1592], 40.00th=[ 1624], 50.00th=[ 1688], 60.00th=[ 1752], | 70.00th=[ 2256], 80.00th=[ 2992], 90.00th=[36608], 95.00th=[38656], | 99.00th=[40704], 99.50th=[41216], 99.90th=[41728], 99.95th=[41728], | 99.99th=[42752] bw (KB/s) : min= 182, max= 769, per=48.49%, avg=504.32, stdev=91.51 lat (usec) : 750=0.01%, 1000=0.09% lat (msec) : 2=66.46%, 4=14.28%, 10=0.83%, 20=0.80%, 50=17.55% cpu : usr=0.43%, sys=2.43%, ctx=196715, majf=65536, minf=27 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0 queryB: (groupid=0, jobs=1): err= 0: pid=10483 read : io=262144KB, bw=517127 B/s, iops=126 , runt=519090msec clat (usec): min=688 , max=47423 , avg=7842.08, stdev=13027.06 lat (usec): min=689 , max=47424 , avg=7842.67, stdev=13027.06 clat percentiles (usec): | 1.00th=[ 1224], 5.00th=[ 1416], 10.00th=[ 1448], 20.00th=[ 1560], | 30.00th=[ 1592], 40.00th=[ 1624], 50.00th=[ 1688], 60.00th=[ 1768], | 70.00th=[ 2256], 80.00th=[ 2992], 90.00th=[36608], 95.00th=[38656], | 99.00th=[40704], 99.50th=[41216], 99.90th=[41728], 99.95th=[41728], | 99.99th=[42240] bw (KB/s) : min= 183, max= 782, per=48.55%, avg=504.87, stdev=87.39 lat (usec) : 750=0.01%, 1000=0.12% lat (msec) : 2=66.05%, 4=14.67%, 10=0.79%, 20=0.79%, 50=17.59% cpu : usr=0.39%, sys=2.10%, ctx=196157, majf=65536, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0 bgupdater: (groupid=0, jobs=1): err= 0: pid=10484 read : io=16164KB, bw=1095.9KB/s, iops=273 , runt= 14751msec slat (usec): min=27 , max=263 , avg=78.80, stdev=26.74 clat (usec): min=574 , max=55711 , avg=16161.04, stdev=15577.04 lat (usec): min=660 , max=55765 , avg=16240.59, stdev=15578.88 clat percentiles (usec): | 1.00th=[ 852], 5.00th=[ 1160], 10.00th=[ 1336], 20.00th=[ 1576], | 30.00th=[ 1992], 40.00th=[ 2864], 50.00th=[ 7648], 60.00th=[20352], | 70.00th=[32128], 80.00th=[35584], 90.00th=[37632], 95.00th=[39168], | 99.00th=[41216], 99.50th=[41728], 99.90th=[45312], 99.95th=[45824], | 99.99th=[55552] bw (KB/s) : min= 838, max= 1512, per=100.00%, avg=1098.41, stdev=164.81 write: io=16604KB, bw=1125.7KB/s, iops=281 , runt= 14751msec slat (usec): min=43 , max=281 , avg=119.68, stdev=33.39 clat (msec): min=5 , max=80 , avg=40.70, stdev= 6.48 lat (msec): min=5 , max=81 , avg=40.82, stdev= 6.48 clat percentiles (usec): | 1.00th=[15296], 5.00th=[29056], 10.00th=[35584], 20.00th=[38144], | 30.00th=[39168], 40.00th=[40192], 50.00th=[41216], 60.00th=[42240], | 70.00th=[43264], 80.00th=[44800], 90.00th=[46848], 95.00th=[48896], | 99.00th=[55040], 99.50th=[57600], 99.90th=[71168], 99.95th=[73216], | 99.99th=[81408] bw (KB/s) : min= 896, max= 1274, per=26.54%, avg=1126.14, stdev=103.62 lat (usec) : 750=0.24%, 1000=1.05% lat (msec) : 2=13.66%, 4=6.97%, 10=4.39%, 20=4.38%, 50=67.46% lat (msec) : 100=1.84% cpu : usr=0.76%, sys=5.91%, ctx=21589, majf=0, minf=22 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=4041/w=4151/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=540452KB, aggrb=1040KB/s, minb=504KB/s, maxb=1095KB/s, mint=14751msec, maxt=519190msec WRITE: io=278748KB, aggrb=4243KB/s, minb=1125KB/s, maxb=3990KB/s, mint=14751msec, maxt=65691msec root@ceph-deploy01:/home/ceph/ceph-cluster# Also i've made the test on a NetApp FAS3250 cluster that we had: root@e05-host05:/home/cloud# fio four-threats.fio bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32 queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1 queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1 bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16 2.0.8 Starting 4 processes bgwriter: Laying out IO file(s) (1 file(s) / 256MB) queryA: Laying out IO file(s) (1 file(s) / 256MB) queryB: Laying out IO file(s) (1 file(s) / 256MB) bgupdater: Laying out IO file(s) (1 file(s) / 32MB) Jobs: 3 (f=3): [wrr_] [2.7% done] [1454K/4075K /s] [363 /1018 iops] [eta 07m:10 Jobs: 3 (f=3): [wrr_] [2.8% done] [359K/4007K /s] [89 /1001 iops] [eta 07m:38s] Jobs: 3 (f=3): [wrr_] [3.4% done] [1162K/3468K /s] [290 /867 iops] [eta 07m:56s Jobs: 3 (f=3): [wrr_] [3.7% done] [1214K/3440K /s] [303 /860 iops] [eta 07m:51s Jobs: 3 (f=3): [wrr_] [3.9% done] [1202K/3676K /s] [300 /919 iops] [eta 07m:47s Jobs: 3 (f=3): [wrr_] [4.1% done] [1194K/3264K /s] [298 /816 iops] [eta 07m:43s Jobs: 3 (f=3): [wrr_] [4.4% done] [1214K/3188K /s] [303 /797 iops] [eta 07m:39s Jobs: 3 (f=3): [wrr_] [4.6% done] [1234K/3080K /s] [308 /770 iops] [eta 07m:35s Jobs: 3 (f=3): [wrr_] [4.9% done] [1254K/3136K /s] [313 /784 iops] [eta 07m:31s Jobs: 3 (f=3): [wrr_] [5.1% done] [1246K/3192K /s] [311 /798 iops] [eta 07m:27s Jobs: 3 (f=3): [wrr_] [5.3% done] [1214K/3172K /s] [303 /793 iops] [eta 07m:23s Jobs: 3 (f=3): [wrr_] [5.6% done] [1238K/3172K /s] [309 /793 iops] [eta 07m:20s Jobs: 3 (f=3): [wrr_] [5.7% done] [535K/3164K /s] [133 /791 iops] [eta 07m:29s] Jobs: 3 (f=3): [wrr_] [6.2% done] [1150K/3100K /s] [287 /775 iops] [eta 07m:37s Jobs: 3 (f=3): [wrr_] [6.4% done] [1262K/3224K /s] [315 /806 iops] [eta 07m:33s Jobs: 3 (f=3): [wrr_] [6.6% done] [1110K/2817K /s] [277 /704 iops] [eta 07m:32s Jobs: 3 (f=3): [wrr_] [6.8% done] [931K/3052K /s] [232 /763 iops] [eta 07m:33s] Jobs: 1 (f=1): [_r__] [99.8% done] [1010K/0K /s] [252 /0 iops] [eta 00m:01s] bgwriter: (groupid=0, jobs=1): err= 0: pid=29603 write: io=262144KB, bw=4563.9KB/s, iops=1140 , runt= 57439msec slat (usec): min=9 , max=24719 , avg=855.55, stdev=1710.97 clat (usec): min=546 , max=281071 , avg=27186.12, stdev=14054.06 lat (usec): min=586 , max=281080 , avg=28042.59, stdev=14380.73 clat percentiles (usec): | 1.00th=[ 1208], 5.00th=[ 4080], 10.00th=[ 7776], 20.00th=[13888], | 30.00th=[19328], 40.00th=[23936], 50.00th=[27776], 60.00th=[31616], | 70.00th=[35584], 80.00th=[39680], 90.00th=[44288], 95.00th=[48384], | 99.00th=[56576], 99.50th=[59648], 99.90th=[72192], 99.95th=[116224], | 99.99th=[257024] bw (KB/s) : min= 2856, max=18376, per=91.18%, avg=4421.46, stdev=2452.38 lat (usec) : 750=0.10%, 1000=0.50% lat (msec) : 2=1.45%, 4=2.82%, 10=8.96%, 20=17.51%, 50=65.07% lat (msec) : 100=3.51%, 250=0.06%, 500=0.01% cpu : usr=0.73%, sys=51.81%, ctx=81663, majf=0, minf=756 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0 queryA: (groupid=0, jobs=1): err= 0: pid=29604 read : io=262144KB, bw=535556 B/s, iops=130 , runt=501227msec clat (usec): min=471 , max=664678 , avg=7580.25, stdev=9256.98 lat (usec): min=472 , max=664679 , avg=7580.55, stdev=9256.99 clat percentiles (usec): | 1.00th=[ 820], 5.00th=[ 996], 10.00th=[ 2064], 20.00th=[ 4256], | 30.00th=[ 5280], 40.00th=[ 6112], 50.00th=[ 6816], 60.00th=[ 7520], | 70.00th=[ 8256], 80.00th=[ 9280], 90.00th=[11584], 95.00th=[14784], | 99.00th=[28800], 99.50th=[41728], 99.90th=[109056], 99.95th=[173056], | 99.99th=[370688] bw (KB/s) : min= 7, max= 1136, per=48.75%, avg=525.52, stdev=141.18 lat (usec) : 500=0.01%, 750=0.25%, 1000=4.82% lat (msec) : 2=4.77%, 4=8.30%, 10=66.28%, 20=13.36%, 50=1.84% lat (msec) : 100=0.26%, 250=0.10%, 500=0.01%, 750=0.01% cpu : usr=0.14%, sys=0.35%, ctx=131440, majf=65536, minf=99 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0 queryB: (groupid=0, jobs=1): err= 0: pid=29605 read : io=262144KB, bw=545259 B/s, iops=133 , runt=492308msec clat (usec): min=366 , max=639108 , avg=7442.54, stdev=8790.57 lat (usec): min=366 , max=639108 , avg=7442.84, stdev=8790.57 clat percentiles (usec): | 1.00th=[ 724], 5.00th=[ 964], 10.00th=[ 1992], 20.00th=[ 4320], | 30.00th=[ 5280], 40.00th=[ 5984], 50.00th=[ 6688], 60.00th=[ 7328], | 70.00th=[ 8096], 80.00th=[ 9024], 90.00th=[11328], 95.00th=[14400], | 99.00th=[28800], 99.50th=[41216], 99.90th=[117248], 99.95th=[171008], | 99.99th=[257024] bw (KB/s) : min= 16, max= 1009, per=49.66%, avg=535.34, stdev=148.02 lat (usec) : 500=0.11%, 750=1.04%, 1000=4.57% lat (msec) : 2=4.30%, 4=7.47%, 10=68.26%, 20=12.12%, 50=1.75% lat (msec) : 100=0.24%, 250=0.12%, 500=0.01%, 750=0.01% cpu : usr=0.14%, sys=0.41%, ctx=131449, majf=65536, minf=101 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0 bgupdater: (groupid=0, jobs=1): err= 0: pid=29606 read : io=16368KB, bw=13918KB/s, iops=3479 , runt= 1176msec slat (usec): min=7 , max=226 , avg=17.68, stdev=13.13 clat (usec): min=569 , max=15737 , avg=2111.01, stdev=985.20 lat (usec): min=588 , max=15812 , avg=2129.16, stdev=985.67 clat percentiles (usec): | 1.00th=[ 1192], 5.00th=[ 1464], 10.00th=[ 1576], 20.00th=[ 1672], | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1848], 60.00th=[ 1960], | 70.00th=[ 2128], 80.00th=[ 2384], 90.00th=[ 2768], 95.00th=[ 3280], | 99.00th=[ 7008], 99.50th=[ 8768], 99.90th=[11968], 99.95th=[12992], | 99.99th=[15680] bw (KB/s) : min=12840, max=14888, per=100.00%, avg=13864.00, stdev=1448.15 write: io=16400KB, bw=13946KB/s, iops=3486 , runt= 1176msec slat (usec): min=8 , max=287 , avg=19.21, stdev=14.16 clat (usec): min=594 , max=13975 , avg=2217.93, stdev=1060.31 lat (usec): min=615 , max=13989 , avg=2237.63, stdev=1060.73 clat percentiles (usec): | 1.00th=[ 1240], 5.00th=[ 1576], 10.00th=[ 1688], 20.00th=[ 1720], | 30.00th=[ 1784], 40.00th=[ 1848], 50.00th=[ 1928], 60.00th=[ 2040], | 70.00th=[ 2224], 80.00th=[ 2448], 90.00th=[ 2832], 95.00th=[ 3440], | 99.00th=[ 7456], 99.50th=[ 9408], 99.90th=[12224], 99.95th=[12992], | 99.99th=[14016] bw (KB/s) : min=12632, max=15224, per=100.00%, avg=13928.00, stdev=1832.82 lat (usec) : 750=0.09%, 1000=0.27% lat (msec) : 2=59.50%, 4=36.61%, 10=3.25%, 20=0.29% cpu : usr=3.74%, sys=11.91%, ctx=8588, majf=0, minf=23 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=4092/w=4100/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=540656KB, aggrb=1078KB/s, minb=523KB/s, maxb=13918KB/s, mint=1176msec, maxt=501227msec WRITE: io=278544KB, aggrb=4849KB/s, minb=4563KB/s, maxb=13945KB/s, mint=1176msec, maxt=57439msec Disk stats (read/write): dm-4: ios=135140/69692, merge=0/2236, ticks=984912/74076, in_queue=1058872, util=100.00%, aggrios=67607/34846, aggrmerge=0/0, aggrticks=490164/35990, aggrin_queue=526150, aggrutil=51.05% sdc: ios=67522/34956, merge=0/0, ticks=489872/37288, in_queue=527156, util=50.86% sdd: ios=67692/34736, merge=0/0, ticks=490456/34692, in_queue=525144, util=51.05% root@e05-host05:/home/cloud# German Anders --- Original message --- |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
