Hello, On Mon, 5 Jun 2017 12:25:25 +0800 TYLin wrote: > Hi all, > > We’re using cache-tier with write-back mode but the write throughput is not as good as we expect. Numbers (what did you see and what did you expect?), versions, cluster HW/SW, etc etc. >We use CephFS and create a 20GB file in it. While data is writing, we use iostat to get the disk statistics. From iostat, we saw that ssd (cache-tier) is idle most of the time and hdd (storage-tier) is busy all the time. From the document While having no real experience with CephFS (with or w/o cache-tiers), I do think I know what you're seeing here, see below. > > “When admins configure tiers with writeback mode, Ceph clients write data to the cache tier and receive an ACK from the cache tier. In time, the data written to the cache tier migrates to the storage tier and gets flushed from the cache tier.” > > So the data is write to cache-tier and then flush to storage tier when dirty ratio is more than 0.4? The word “in time” in the document confused me. > > We found that the throughput of creating a new file is slower than overwrite an existing file, and ssd has more write when doing overwrite. We then look into the source code and log. A newly created file goes to proxy_write, which is followed by a promote_object. Does this means that the object actually goes to storage pool directly and then be promoted to the cache-tier when creating a new file? > Creating a new file means creating new Ceph objects, which need to be present on both the backing store and the cache-tier. That overhead of creating them is the difference in time you see. The actual data of the initial write will still be only on the cache-tier, btw. Once a file exists and is properly (not sparsely) allocated, writes should indeed just go to the cache-tier until flushing (space/time/object#) becomes necessary. That of course also requires the cache being big enough and not too busy so that things stay actually in it. Otherwise those objects need to be promoted back in from the HDDs, making things slow again. Tuning a cache-tier (both parameters and size in general) isn't easy and with some workloads pretty impossible to get desirable results. Christian -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Rakuten Communications _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |