Hey Felix, I run bcache pretty much in the way you're describing, but we have smaller spinning disks (4TB). We mostly share a 1TB NVMe between 6x osd's with 33GB db/wal per osd and the rest shared bcache cache. The performance is definitely improved over not running cache. We run this mostly for rbd replicated pool for vm disk. As I learnt when discussing bcache in another thread here it's important to set rotational in sysfs for the device (e.g. /sys/devices/virtual/block/bcache0/queue/rotational) before creating the osd, otherwise ceph detects it as SSD storage and will default a bunch of parameters that may not be appropriate. I was able to manually set some stuff back and improve latency but I think for my setup and possibly yours the performance would be improved with ceph seeing it as fast spinning rather than slow SSD/NVMe. The reason I've done it the way you have also suggested with separate db/wal rather than what Burkhard has done with db/wal on the cached disk is that I want the db/wal to always be in the cache for all osds and I've also set the bluestore_prefer_deferred_size so that it's passing small i/o through the wal, which is the default for spinning disks. For me this gives the lowest latency, with the caveat that it has multiple writes due to wal->bcache->spinning. Statistically the db / wal should always be in the cache due to constant read/write and as Burkhard said it may waste space but I feel it's an ok compromise for consistency. Regards, Rich > > Hey guys, > > > > i have three servers with 12x 12 TB Sata HDDs and 1x 3,4 TB NVME. I am thinking of putting DB/WAL on the NVMe as well as an 5GB DM-Cache for each spinning disk. Is anyone running something like this in a production environment? _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |