Hi Everyone, I just wanted to share a discovery I made about running bluestore on top of Bcache in case anyone else is doing this or considering it. We've run Bcache under Filestore for a long time with good results but recently rebuilt all the osds on bluestore. This caused some degradation in performance that I couldn't quite put my finger on. Bluestore osds have some smarts where they detect the disk type. Unfortunately in the case of Bcache it detects as SSD, when in fact the HDD parameters are better suited. I changed the following parameters to match the HDD default values and immediately saw my average osd latency during normal workload drop from 6ms to 2ms. Peak performance didn't change really, but a test machine that I have running a constant iops workload was much more stable as was the average latency. Performance has returned to Filestore or better levels. Here are the parameters. ; Make sure that we use values appropriate for HDD not SSD - Bcache gets detected as SSD bluestore_prefer_deferred_size = 32768 bluestore_compression_max_blob_size = 524288 bluestore_deferred_batch_ops = 64 bluestore_max_blob_size = 524288 bluestore_min_alloc_size = 65536 bluestore_throttle_cost_per_io = 670000 ; Try to improve responsiveness when some disks are fully utilised osd_op_queue = wpq osd_op_queue_cut_off = high Hopefully someone else finds this useful. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |