Great Work! This is very exciting! Did you happen to try RADOS bench at
different object sizes and concurrency levels?
Mark
On 11/24/2013 03:01 AM, Haomai Wang wrote:
Hi all,
For Emperor
blueprint(http://wiki.ceph.com/01Planning/02Blueprints/Emperor/Add_LevelDB_support_to_ceph_cluster_backend_store),
I'm sorry to delay the progress. Now, I have done the most of the works
for the blueprint's goal. Because of sage's F
blueprint(http://wiki.ceph.com/index.php?title=01Planning/02Blueprints/Firefly/osd:_new_key%2F%2Fvalue_backend),
I need to adjust some codes to match it. The branch is
here(https://github.com/yuyuyu101/ceph/tree/wip/6173).
I have tested the LevelDB backend on three nodes(eight OSDs) and compare
it to FileStore(ext4). I just use intern benchmark tool "rados bench" to
get the comparison. The default ceph configurations is used and
replication size is 2. The filesystem is ext4 and no others changed. The
results is below:
*Rados Bench*
*Bandwidth(MB/sec)*
*Average Latency*
*Max Latency*
*Min Latency*
*Stddev Latency*
*Stddev Bandwidth(MB/sec)*
*Max Bandwidth(MB/sec)*
*Min Bandwidth(MB/sec)*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*Write 30*
24.590
23.495
4.87257
5.07716
14.752
13.0885
0.580851
0.605118
2.97708
3.30538
9.91938
10.5986
44
76
0
0
*Write 20*
23.515
23.064
3.39745
3.45711
11.6089
11.5996
0.169507
0.138595
2.58285
2.75962
9.14467
8.54156
44
40
0
0
*Write 10*
22.927
21.980
1.73815
1.8198
5.53792
6.46675
0.171028
0.143392
1.05982
1.20303
9.18403
8.74401
44
40
0
0
*Write 5*
19.680
20.017
1.01492
0.997019
3.10783
3.05008
0.143758
0.138161
0.561548
0.571459
5.92575
6.844
36
32
0
0
*Read 30*
65.852
60.688
1.80069
1.96009
9.30039
10.1146
0.115153
0.061657
*Read 20*
59.372
60.738
1.30479
1.28383
6.28435
8.21304
0.016843
0.012073
*Read 10*
65.502
55.814
0.608805
0.7087
3.3917
4.72626
0.016267
0.011998
*Read 5*
64.176
54.928
0.307111
0.364077
1.76391
1.90182
0.017174
0.011999
Charts can be view here(http://img42.com/ziwjP+) and
(http://img42.com/LKhoo+)
From above, I'm feeling relieved that the LevelDB backend isn't
useless. Most of metrics are better and if increasing cache size for
LevelDB the results may be more attractive.
Even more, LevelDB backend is used by "KeyValueStore" and much of
optimizations can be done to improve performance such as increase
parallel threads or optimize io path.
Next, I use "rbd bench-write" to test. The result is pity:
*RBD Bench-Write*
*OPS/sec*
*Bytes/sec*
*KVStore*
*FileStore*
*KVStore*
*FileStore*
*Seq 4096 5*
27.42
716.55
111861.51
2492149.21
*Rand 4096 5*
28.27
504
112331.42
1683151.29
Just because kv backend doesn't support read/write operation with
offset/length argument, each read/write operation will call a additional
read LevelDB api to do. Much of time is consumed by reading entire large
object in rbd situation. There exists some ways to change such as split
large object to multi small objects or save metadata to avoid read
arduous operation.
As sage mentioned in <osd: new key/value
backend>(http://wiki.ceph.com/index.php?title=01Planning/02Blueprints/Firefly/osd:_new_key%2F%2Fvalue_backend),
more kv backends can be added now and I look forward to more people
interested it. I think radosgw situation can fit in kv store in short ti
--
Best Regards,
Wheat
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
