Hi Alexandre, below is random io test results, almost the same iops.
Rocksdb results
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64
fio-2.1.4
Starting 1 thread
rbd engine: RBD version: 0.1.8
Jobs: 1 (f=1): [w] [100.0% done] [0KB/23094KB/0KB /s] [0/5773/0 iops] [eta 00m:00s]
ebs_test: (groupid=0, jobs=1): err= 0: pid=47154: Tue Mar 4 13:48:22 2014
write: io=3356.2MB, bw=17183KB/s, iops=4295, runt=200004msec
slat (usec): min=19, max=8855, avg=134.33, stdev=259.00
clat (usec): min=73, max=4397.6K, avg=12756.12, stdev=79341.35
lat (msec): min=1, max=4397, avg=12.89, stdev=79.34
clat percentiles (usec):
| 1.00th=[ 1432], 5.00th=[ 1752], 10.00th=[ 2128], 20.00th=[ 3408],
| 30.00th=[ 4768], 40.00th=[ 5856], 50.00th=[ 6880], 60.00th=[ 7904],
| 70.00th=[ 8896], 80.00th=[10048], 90.00th=[11968], 95.00th=[14016],
| 99.00th=[27520], 99.50th=[505856], 99.90th=[1204224], 99.95th=[1433600],
| 99.99th=[2834432]
bw (KB /s): min= 403, max=24392, per=100.00%, avg=17358.47, stdev=7446.69
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=8.36%, 4=15.77%, 10=55.27%, 20=19.17%, 50=0.51%
lat (msec) : 100=0.09%, 250=0.16%, 500=0.14%, 750=0.19%, 1000=0.15%
lat (msec) : 2000=0.16%, >=2000=0.01%
cpu : usr=18.04%, sys=4.15%, ctx=1875119, majf=0, minf=838
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=1.1%, 16=10.9%, 32=65.9%, >=64=22.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.6%, 8=0.4%, 16=0.4%, 32=0.6%, 64=0.9%, >=64=0.0%
issued : total=r=0/w=859165/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=3356.2MB, aggrb=17182KB/s, minb=17182KB/s, maxb=17182KB/s, mint=200004msec, maxt=200004msec
Disk stats (read/write):
sda: ios=0/2191, merge=0/2904, ticks=0/936, in_queue=936, util=0.29%
leveldb results:
ebs_test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=64
fio-2.1.4
Starting 1 thread
rbd engine: RBD version: 0.1.8
Jobs: 1 (f=1): [w] [100.0% done] [0KB/9428KB/0KB /s] [0/2357/0 iops] [eta 00m:00s]
ebs_test: (groupid=0, jobs=1): err= 0: pid=112425: Tue Mar 4 14:54:00 2014
write: io=3404.9MB, bw=17431KB/s, iops=4357, runt=200016msec
slat (usec): min=20, max=7698, avg=114.01, stdev=201.06
clat (usec): min=220, max=3278.3K, avg=13340.59, stdev=76874.35
lat (msec): min=1, max=3278, avg=13.45, stdev=76.87
clat percentiles (usec):
| 1.00th=[ 1400], 5.00th=[ 1608], 10.00th=[ 1784], 20.00th=[ 2192],
| 30.00th=[ 2832], 40.00th=[ 3824], 50.00th=[ 5024], 60.00th=[ 6240],
| 70.00th=[ 7456], 80.00th=[ 8768], 90.00th=[10816], 95.00th=[13120],
| 99.00th=[284672], 99.50th=[610304], 99.90th=[1089536], 99.95th=[1286144],
| 99.99th=[1630208]
bw (KB /s): min= 24, max=25548, per=100.00%, avg=17606.69, stdev=6779.23
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=15.63%, 4=25.94%, 10=45.35%, 20=10.98%, 50=0.44%
lat (msec) : 100=0.17%, 250=0.40%, 500=0.42%, 750=0.34%, 1000=0.19%
lat (msec) : 2000=0.12%, >=2000=0.01%
cpu : usr=18.25%, sys=4.14%, ctx=1887389, majf=0, minf=742
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.5%, 16=6.0%, 32=55.9%, >=64=37.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.8%, 8=0.7%, 16=0.5%, 32=0.5%, 64=0.5%, >=64=0.0%
issued : total=r=0/w=871635/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=3404.9MB, aggrb=17431KB/s, minb=17431KB/s, maxb=17431KB/s, mint=200016msec, maxt=200016msec
Disk stats (read/write):
sda: ios=0/2125, merge=0/2796, ticks=0/708, in_queue=708, util=0.23%
-----Original Message-----
From: Alexandre DERUMIER [mailto:aderumier@xxxxxxxxx]
Sent: Tuesday, March 04, 2014 12:49 PM
To: Shu, Xinxin
Cc: ceph-devel@xxxxxxxxxxxxxxx
Subject: Re: [RFC] add rocksdb support
>>Performance Test
>>Attached file is the performance comparison of rocksdb and leveldb on four nodes with 40 osds, using 'rados bench' as the test tool. The performance results is quite promising.
Thanks for your work, indeed performance seem to be promising !
>>Any comments or suggestions are greatly appreciated.
Could you do test with random io write with last fio (with rbd support) ?
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-February/008182.html
> The fio command: fio -direct=1 -iodepth=64 -thread -rw=randwrite
>> -ioengine=rbd -bs=4k -size=19G -numjobs=1 -runtime=100
>> -group_reporting -name=ebs_test -pool=openstack -rbdname=image
>> -clientname=fio -invalidate=0
----- Mail original -----
De: "Xinxin Shu" <xinxin.shu@xxxxxxxxx>
À: ceph-devel@xxxxxxxxxxxxxxx
Envoyé: Lundi 3 Mars 2014 03:07:18
Objet: [RFC] add rocksdb support
Hi all,
This patch added rocksdb support for ceph, enabled rocksdb for omap directory. Rocksdb source code can be get from link. To use use rocksdb, C++11 standard should be enabled, gcc version >= 4.7 is required to get C++11 support. Rocksdb can be installed with instructions described in the INSTALL.md file, and rocksdb header files (include/rocksdb/*) and library (librocksdb.so*) need to be copied to corresponding directories.
To enable rocksdb, add "--with-librocksdb" option to configure. The rocksdb branch is here(https://github.com/xinxinsh/ceph/tree/rocksdb).
Performance Test
Attached file is the performance comparison of rocksdb and leveldb on four nodes with 40 osds, using 'rados bench' as the test tool. The performance results is quite promising.
Any comments or suggestions are greatly appreciated.
Rados bench BandWidth(MB/s) Average latency Leveldb rocksdb Leveldb rocksdb write 4 threads 263.762 272.549 0.061 0.059 write 8 threads 449.834 457.811 0.071 0.070 write 16 threads 642.100 638.972 0.100 0.100 write 32 threads 705.897 717.598 0.181 0.178 write 64 threads 705.011 717.204 0.370 0.362 read 4 threads 873.588 841.704 0.073 0.076 read 8 threads 816.699 818.451 0.078 0.078 read 16 threads 808.810 798.053 0.079 0.080 read 32 threads 798.394 802.796 0.080 0.080 read 64 threads 792.848 790.593 0.081 0.081
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
