On 08/18/2015 11:52 AM, Nick Fisk wrote:
<snip>
Here's kind of how I see the field right now:
1) Cache at the client level. Likely fastest but obvious issues like above.
RAID1 might be an option at increased cost. Lack of barriers in some
implementations scary.
Agreed.
2) Cache below the OSD. Not much recent data on this. Not likely as
fast as client side cache, but likely cheaper (fewer OSD nodes than client
nodes?).
Lack of barriers in some implementations scary.
This also has the benefit of caching the leveldb on the OSD, so get a big
performance gain from there too for small sequential writes. I looked at
using Flashcache for this too but decided it was adding to much complexity
and risk.
I thought I read somewhere that RocksDB allows you to move its WAL to
SSD, is there anything in the pipeline for something like moving the filestore
to use RocksDB?
I believe you can already do this, though I haven't tested it. You can certainly
move the monitors to rocksdb (tested) and newstore uses rocksdb as well.
Interesting, I might have a look into this.
3) Ceph Cache Tiering. Network overhead and write amplification on
promotion makes this primarily useful when workloads fit mostly into the
cache tier. Overall safe design but care must be taken to not over-
promote.
4) separate SSD pool. Manual and not particularly flexible, but perhaps
best
for applications that need consistently high performance.
I think it depends on the definition of performance. Currently even very
fast CPU's and SSD's in their own pool will still struggle to get less than 1ms of
write latency. If your performance requirements are for large queue depths
then you will probably be alright. If you require something that mirrors the
performance of traditional write back cache, then even pure SSD Pools can
start to struggle.
Agreed. This is definitely the crux of the problem. The example below
is a great start! It'd would be fantastic if we could get more feedback
from the list on the relative importance of low latency operations vs
high IOPS through concurrency. We have general suspicions but not a ton
of actual data regarding what folks are seeing in practice and under
what scenarios.
If you have any specific questions that you think I might be able to answer, please let me know. The only other main app that I can really think of where these sort of write latency is critical is SQL, particularly the transaction logs.
Probably the big question is what are the pain points? The most common
answer we get when asking folks what applications they run on top of
Ceph is "everything!". This is wonderful, but not helpful when trying
to figure out what performance issues matter most! :)
IE, should we be focusing on IOPS? Latency? Finding a way to avoid
journal overhead for large writes? Are there specific use cases where
we should specifically be focusing attention? general iscsi? S3?
databases directly on RBD? etc. There's tons of different areas that we
can work on (general OSD threading improvements, different messenger
implementations, newstore, client side bottlenecks, etc) but all of
those things tackle different kinds of problems.
Mark
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
