Hi, I also have used bcache extensively on filestore with journals on SSD for at least 5 years. This has worked very well in all versions up to luminous. The iops improvement was definitely beneficial for vm disk images in rbd. I am also using it under bluestore with db/wal on nvme on both Luminous and Nautilus. This is a smaller cluster also for vm disk images on rbd, and the iops appeared to be improved. However as Matthias mentioned, be sure to get the right types of SSD's as you can have strange performance issues. Make sure you do fio testing as I found on paper Intel DC S4600 looked ok but performed very badly in sequential 4k writes. Also keep an eye on your drive writes per day as in a busy cluster you may hit 1DWPD or more. Stability has been very good with bcache. I'm using it on Ubuntu and have had to use newer kernels on some of the historical LTS releases. In 16.04 the 4.15 edge kernel is recommended, newer versions stock kernel is fine. Rich On Tue, 20 Apr 2021 at 03:37, Matthias Ferdinand <mf+ml.ceph@xxxxxxxxx> wrote: > > On Sun, Apr 18, 2021 at 10:31:30PM +0200, huxiaoyu@xxxxxxxxxxxx wrote: > > Dear Cephers, > > > > Just curious about any one who has some experience on using Bcache on top of HDD OSD to accelerate IOPS performance? > > > > If any, how about the stability and the performance improvement, and for how long the running time? > > Hi, > > I have not used bcache with Bluestore, but I use bcache in a Jewel > cluster with Filestore on XFS on bcache on HDD, and I haven't seen any > bcache-related trouble with this setup so far. I don't have journal on > bcache, journal is separated out to SSD. > From time to time I drained and detached the caching device from an OSD > just to see if the added complexity still has some value, but latency > (as measured by iostat on the OSD) would go up by a factor of about 2 so > I kept bcache active on HDD OSDs. > > For Bluestore I don't know how much (if at all) bcache would improve > performance where WAL/DB is placed on SSD already. > > A word of warning: never use non-DC-class SSDs for bcache caching > devices. bcache does some write amplification, and you will regret it if > using consumer grade SSDs. Might even turn out slower than plain HDD > with highly irregular latency spikes. > > > Regards > Matthias > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |