I Have a setup using some Intel P3700 devices as a cache tier, and 33 sata drives hosting the pool behind them. I setup the cache tier with writeback, gave it a size and max object count etc: ceph osd pool set target_max_bytes 500000000000 ceph osd pool set nvme target_max_bytes 500000000000 ceph osd pool set nvme target_max_objects 500000 ceph osd pool set nvme cache_target_dirty_ratio 0.5 ceph osd pool set nvme cache_target_full_ratio 0.8 This is all running Jewel using bluestore OSDs (I know experimental). The cache tier will write at about 900 Mbytes/sec and read at 2.2 Gbytes/sec, the sata pool can take writes at about 600 Mbytes/sec in aggregate. However, it looks like the mechanism for cleaning the cache down to the disk layer is being massively rate limited and I see about 47 Mbytes/sec of read activity from each SSD while this is going on. This means that while I could be pushing data into the cache at high speed, It cannot evict old content very fast at all, and it is very easy to hit the high water mark and the application I/O drops dramatically as it becomes throttled by how fast the cache can flush. I suspect it is operating on a placement group at a time so ends up targeting a very limited number of objects and hence disks at any one time. I can see individual disk drives going busy for very short periods, but most of them are idle at any one point in time. The only way to drive the disk based OSDs fast is to hit a lot of them at once which would mean issuing many cache flush operations in parallel. Are there any controls which can influence this behavior? Thanks Steve ---------------------------------------------------------------------- The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |