Hello, On Fri, 28 Oct 2016 12:42:40 +0800 (CST) james wrote: > Hi, > > Is there anyone in the community has experience of using "bcache" as backend of Ceph? If you search the ML archives (via google) you should find some past experiences, more recent ones not showing particular great improvements over journal SSD. Also read the current "cache tiering deprecated in RHCS 2.0" thread, the release notes state that the Ceph developers are looking at dm-cache. > Nowadays, maybe most Ceph solution are based on full-SSD or full-HDD as backend data disks. So in order > to balance the cost and performance/capacity, we are trying the hybrid solution with "bcache". It utilize the SSD as cache > of the HDD to improve the performance(especially for random IO) and also easy to implement as it is already included in linux kernel. > Not really, most people tend to use HDD based OSDs with SSD journals. > https://en.wikipedia.org/wiki/Bcache > https://github.com/torvalds/linux/tree/master/drivers/md/bcache > > Now I have 2 queries below and hope the storage expert in this community can kindly share some suggestions: > 1. Is there any best practise of bcache tuning for using in ceph? > - Currently we enable the "writeback" mode of bcache and leave the other setting as default > - bcache has a writeback throttling algrithm to control the rate of SSD cache-> HDD writeback > > > 2. Since we are running a Ceph cluster based on a storage backend with a "cache" layer, how to design a benchmark scenario that can get a stable/predictable IOPS result? Caches can only be predictable until they are overwhelmed. A 4GB HW cache on an Areca raid controller can do wonders for IOPS, but after a certain point or with certain I/O patterns it will be limited by what the HDDs behind it can do natively. If you need a certain performance in any situation and all the time your base storage layer must be able to provide it, no matter if 99% of the time any cache layers above it will give you vastly superior results. > - We use the ceph as backend of block storage for openstack cinder service. Our benchmark is running in openstack guest VMs against the attached volume(block storage) in ceph > - In our test, we noticed that the read/write IOPS highly depends on the bcache cache hit rate and the amount of data to be write compared with "dirty_data" threshold specified in the bcache configuration > 1) If we run read test for several round repeatedly, as more and more data can be cached in the SSD layer, the IOPS result will gradually increase due to increasing cache hit rate > 2) If we run write test with large amount of data, when the size of dirty_data in the SSD cache layer reach the threshold, it will began to writeback/flush to the backend HDD, and then the incoming IOPS will drop down or fluctuate > The MQ mode of dm-cache supposedly works around this issue by doing large sequential operations against the HDDs and small random OPS against the SSDs. Christian > - I suppose besides bcache, Ceph has other solution with "cache tier/layer", how to benchmark under this kind of design? Because if we can not find a scenario to get a stable benchmark IOPS result, we can not evaluate the impact of configuration/code change of ceph on the ceph performance (sometimes IOPS result increase/drop may only due to bcache behavior) > > Thank you very much and looking forward to your expertise suggestion! -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |