Hi, Sage Currently, we find that separate the block.db and wal to SSD cannot significantly achieve the performance improvement. Especially for those deferred IO, the main reason is caused by the force_flush in _kv_sync_thread. In _kv_sync_thread, it wants to flush all previous committed IO to the hdd disk and then cleanup the wal. The main problem is that it is a single synchronize _kv_sync_thread to commit all IOs waited to be committed, so lots of IO need wait the force_flush and, the client latency is quite poor under this situation, which also affects the burst IO like FileStore with SSD Journal. I wonder if it is possible to make the force_flush and wal cleanup process asynchronously so that we can achieve the burst write IO with low/stable latency? I remember, at the beginning, it is asynchronous, right? Regards Ning Yao 2018-07-14 9:46 GMT+08:00 shuhao li <letterwuyu@xxxxxxxxx>: > Hi all, > > I tested the performance of bluestore in a mixed scene of hdd and ssd. > There are three nodes here, each node has 1 mon and 1 mgr 12 osd， > Each osd has 1.7T HDD for block, 150G SSD partition for db and wal, 10Gib bond. > > system： CentOS Linux (3.10.0-862.3.2.el7.x86_64) 7 (Core) > ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) > > I tested that the performance is very poor and the delay is hug. > I found that _kv_sync_thread force_flush took a long time to block the kv commit. > so 4krandwrite 32dp IOPS=2469, BW=9878KiB/s lat 12955.49 us. > > I have some thinking about mixed scene of hdd and ssd： > Hdd fdatasync is slow，Can we avoid fdatasync on the wite path（_kv_sync_thread）? > batch sync to hdd in the background，For deferred_write can use wal for caching， > Is it possible to do a cache on ssd for simple write? > > Have we had a discussion about the built-in ssd cache for bluestore?-- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
