Hi, Am 28.02.2014 03:45, schrieb Haomai Wang: [...] > I use fio which rbd supported from > TelekomCloud(https://github.com/TelekomCloud/fio/commits/rbd-engine) > to test rbd. I would recommend to no longer use this branch, it's outdated. The rbd engine got contributed back to upstream fio and is now merged [1]. For more information read [2]. [1] https://github.com/axboe/fio/commits/master [2] http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html > > The fio command: fio -direct=1 -iodepth=64 -thread -rw=randwrite > -ioengine=rbd -bs=4k -size=19G -numjobs=1 -runtime=100 > -group_reporting -name=ebs_test -pool=openstack -rbdname=image > -clientname=fio -invalidate=0 Don't use runtime and size at the same time, since runtime limits the size. What we normally do we let the fio job fill up the whole rbd or we limit it only via runtime. > ============================================ > > FileStore result: > ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64 > fio-2.1.4 > Starting 1 thread > rbd engine: RBD version: 0.1.8 > > ebs_test: (groupid=0, jobs=1): err= 0: pid=30886: Thu Feb 27 08:09:18 2014 > write: io=283040KB, bw=6403.4KB/s, iops=1600, runt= 44202msec > slat (usec): min=116, max=2817, avg=195.78, stdev=56.45 > clat (msec): min=8, max=661, avg=39.57, stdev=29.26 > lat (msec): min=9, max=661, avg=39.77, stdev=29.25 > clat percentiles (msec): > | 1.00th=[ 15], 5.00th=[ 20], 10.00th=[ 23], 20.00th=[ 28], > | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 37], 60.00th=[ 40], > | 70.00th=[ 43], 80.00th=[ 46], 90.00th=[ 51], 95.00th=[ 58], > | 99.00th=[ 128], 99.50th=[ 210], 99.90th=[ 457], 99.95th=[ 494], > | 99.99th=[ 545] > bw (KB /s): min= 2120, max=12656, per=100.00%, avg=6464.27, stdev=1726.55 > lat (msec) : 10=0.01%, 20=5.91%, 50=83.35%, 100=8.88%, 250=1.47% > lat (msec) : 500=0.34%, 750=0.05% > cpu : usr=29.83%, sys=1.36%, ctx=84002, majf=0, minf=216 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=17.4%, >=64=82.6% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=99.1%, 8=0.5%, 16=0.3%, 32=0.1%, 64=0.1%, >=64=0.0% > issued : total=r=0/w=70760/d=0, short=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=64 > > Run status group 0 (all jobs): > WRITE: io=283040KB, aggrb=6403KB/s, minb=6403KB/s, maxb=6403KB/s, > mint=44202msec, maxt=44202msec > > Disk stats (read/write): > sdb: ios=5/9512, merge=0/69, ticks=4/10649, in_queue=10645, util=0.92% > > =============================================== > > KeyValueStore: > ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64 > fio-2.1.4 > Starting 1 thread > rbd engine: RBD version: 0.1.8 > > ebs_test: (groupid=0, jobs=1): err= 0: pid=29137: Thu Feb 27 08:06:30 2014 > write: io=444376KB, bw=6280.2KB/s, iops=1570, runt= 70759msec > slat (usec): min=122, max=3237, avg=184.51, stdev=37.76 > clat (msec): min=10, max=168, avg=40.57, stdev= 5.70 > lat (msec): min=11, max=168, avg=40.75, stdev= 5.71 > clat percentiles (msec): > | 1.00th=[ 34], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 39], > | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 41], > | 70.00th=[ 42], 80.00th=[ 42], 90.00th=[ 44], 95.00th=[ 45], > | 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 163], 99.95th=[ 167], > | 99.99th=[ 167] > bw (KB /s): min= 4590, max= 7480, per=100.00%, avg=6285.69, stdev=374.22 > lat (msec) : 20=0.02%, 50=99.58%, 100=0.23%, 250=0.17% > cpu : usr=29.11%, sys=1.10%, ctx=118564, majf=0, minf=194 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.7%, >=64=99.3% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0% > issued : total=r=0/w=111094/d=0, short=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=64 > > Run status group 0 (all jobs): > WRITE: io=444376KB, aggrb=6280KB/s, minb=6280KB/s, maxb=6280KB/s, > mint=70759msec, maxt=70759msec > > Disk stats (read/write): > sdb: ios=0/15936, merge=0/272, ticks=0/17157, in_queue=17146, util=0.94% > > > It's just a simple test, maybe exist some misleadings on the config or > results. But > we can obviously see the conspicuous improvement for KeyValueStore. The numbers are hard to compare. Since the tests wrote a different amount of data. This could influence the numbers. Do you mean improvements compared to former implementation or to FileStore? Without a retest with the latest fio rbd engine: there is not so much difference between KVS and FS atm. Btw. Nice to see the rbd engine is useful to others ;-) Regards Danny -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
