Still I wonder can we flush the deferred io to the bdev asynchronously out of the function _kv_sync_thread()? https://github.com/ceph/ceph/blob/master/src/os/bluestore/BlueStore.cc#L10071 The reason might not do it hope to delete the wal kv pair in bluefs as soon as possible so that the wal would not compact to the other level (such as level 1 or 2)? Otherwise asynchronously flush the dirty data would be much more effiecient. anyone can explain that? Regards Ning Yao Ning Yao <zay11022@xxxxxxxxx> 于2018年7月18日周三 下午6:26写道： > > Hi, Sage > > Currently, we find that separate the block.db and wal to SSD cannot > significantly achieve the performance improvement. Especially for > those deferred IO, the main reason is caused by the force_flush in > _kv_sync_thread. > In _kv_sync_thread, it wants to flush all previous committed IO to the > hdd disk and then cleanup the wal. The main problem is that it is a > single synchronize _kv_sync_thread to commit all IOs waited to be > committed, so lots of IO need wait the force_flush and, the client > latency is quite poor under this situation, which also affects the > burst IO like FileStore with SSD Journal. > I wonder if it is possible to make the force_flush and wal cleanup > process asynchronously so that we can achieve the burst write IO with > low/stable latency? I remember, at the beginning, it is asynchronous, > right? > Regards > Ning Yao > > > 2018-07-14 9:46 GMT+08:00 shuhao li <letterwuyu@xxxxxxxxx>: > > Hi all, > > > > I tested the performance of bluestore in a mixed scene of hdd and ssd. > > There are three nodes here, each node has 1 mon and 1 mgr 12 osd， > > Each osd has 1.7T HDD for block, 150G SSD partition for db and wal, 10Gib bond. > > > > system： CentOS Linux (3.10.0-862.3.2.el7.x86_64) 7 (Core) > > ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) > > > > I tested that the performance is very poor and the delay is hug. > > I found that _kv_sync_thread force_flush took a long time to block the kv commit. > > so 4krandwrite 32dp IOPS=2469, BW=9878KiB/s lat 12955.49 us. > > > > I have some thinking about mixed scene of hdd and ssd： > > Hdd fdatasync is slow，Can we avoid fdatasync on the wite path（_kv_sync_thread）? > > batch sync to hdd in the background，For deferred_write can use wal for caching， > > Is it possible to do a cache on ssd for simple write? > > > > Have we had a discussion about the built-in ssd cache for bluestore?-- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html
