Could you show your cache tiering configuration? Especially this three parameters.
ceph osd pool set hot-storage cache_target_dirty_ratio 0.4
ceph osd pool set hot-storage cache_target_full_ratio 0.8
ceph osd pool set {cachepool} target_max_bytes {#bytes}
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Andrei Mikhailovsky
Sent: Wednesday, September 10, 2014 8:51 PM
To: ceph-users
Subject: Cache Pool writing too much on ssds, poor performance?
Hello guys,
I am experimeting with cache pool and running some tests to see how adding the cache pool improves the overall performance of our small cluster.
While doing testing I've noticed that it seems that the cache pool is writing too much on the cache pool ssds. Not sure what the issue here, perhaps someone could help me understand what is going on.
My test cluster is:
2 x OSD servers (Each server has: 24GB ram, 12 cores, 8 hdd osds, 2 ssds journals, 2 ssds for cache pool, 40gbit/s infiniband network capable of 25gbit/s over ipoib). Cache pool is set to 500GB with replica of 2.
4 x host servers (128GB ram, 24 core, 40gbit/s infiniband network capable of 12gbit/s over ipoib)
So, my test is:
Simple tests using the following command: "dd if=/dev/vda of=/dev/null bs=4M count=2000 iflag=direct". I am concurrently starting this command on 10 virtual machines which are running on 4 host servers. The aim is to monitor the use of cache pool when reading the same data over and over again.
Running the above command for the first time does what I was expecting. The osds are doing a lot of reads, the cache pool does a lot of writes (around 250-300MB/s per ssd disk) and no reads. The dd results for the guest vms are poor. The results of the "ceph -w" shows consistent performance across the time.
Running the above for the second and consequent times produces IO patterns which I was not expecting at all. The hdd osds are not doing much (this part I expected), the cache pool still does a lot of writes and very little reads! The dd results have improved just a little, but not much. The results of the "ceph -w" shows performance breaks over time. For instance, I have a peak of throughput in the first couple of seconds (data is probably coming from the osd server's ram at high rate). After the peak throughput has finished, the ceph reads are done in the following way: 2-3 seconds of activity followed by 2 seconds if inactivity) and it keeps doing that throughout the length of the test. So, to put the numbers in perspective, when running tests over and over again I would get around 2000 - 3000MB/s for the first two seconds, followed by 0MB/s for the next two seconds, followed by around 150-250MB/s over 2-3 seconds, followed by 0MB/s for 2 seconds, followed 150-250MB/s over 2-3 seconds, followed by 0MB/s over 2 secods, and the pattern repeats until the test is done.
I kept running the dd command for about 15-20 times and observed the same behariour. The cache pool does mainly writes (around 200MB/s per ssd) when guest vms are reading the same data over and over again. There is very little read IO (around 20-40MB/s). Why am I not getting high read IO? I have expected the 80GB of data that is being read from the vms over and over again to be firmly recognised as the hot data and kept in the cache pool and read from it when guest vms request the data. Instead, I mainly get writes on the cache pool ssds and I am not really sure where these writes are coming from as my hdd osds are being pretty idle.
