Re: ceph & hbase:

[ker can <kercan74@xxxxxxxxx>]

how is your hbase performance on ceph compared to hdfs  - was there some special knobs that you needed to turn ?

I'm running a few hbase tests with the yahoo cloud serving benchmark - ycsb  with ceph  & hdfs and the results were very surprising considering that the hadoop + ceph results were not all that bad.  For the test setup, I have about 50 million rows distributed over two nodes. Ran a single ycsb client  with 128 threads reading all that doing 100% reads.

the hbase+hdfs throughput results were 38x better. Looking at the system stats I don't see anything special going on - no significant cpu utilization. Practically no disk reads which means its all cached somewhere. The 10gig network is no where close to saturating. Plenty of memory also. 

Any thoughts on what might be going on ?

For hbase + hdfs:
-------------------------
[OVERALL], RunTime(ms), 720023.0
[OVERALL], Throughput(ops/sec), 16373.007528926159
[READ], Operations, 11788942
[READ], AverageLatency(us), 7814.152212217178
[READ], MinLatency(us), 166
[READ], MaxLatency(us), 596026
[READ], 95thPercentileLatency(ms), 19
[READ], 99thPercentileLatency(ms), 23
[READ], Return=0, 11788942

For hbase + ceph:
--------------------------

[OVERALL], RunTime(ms), 720681.0
[OVERALL], Throughput(ops/sec), 420.35102909609105
[READ], Operations, 302939
[READ], AverageLatency(us), 304294.1931676014
[READ], MinLatency(us), 312
[READ], MaxLatency(us), 1539757
[READ], 95thPercentileLatency(ms), 676
[READ], 99thPercentileLatency(ms), 931
[READ], Return=0, 302939

Thx

On Wed, Jul 17, 2013 at 6:27 PM, Mike Bryant <mike.bryant@xxxxxxxxx> wrote:

Yup, that was me.
We have hbase working here.
You'll want to disable localized reads, as per bug #5388. That bug
will cause your regionservers to crash fairly often when doing
compaction.
You'll also want to restart each of the regionservers and masters
often (We're doing it once a day) to mitigate the effects of bug
#5039, being that your data pool is growing much faster than you might
expect, and it being much larger than the visible filesize in cephfs.

With those workarounds in place we're running a stable install of
openTSDB on top of hbase.

Mike

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com