Re: Cache tier operation clarifications

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I'd like to get some insights, confirmations from people here who are
either familiar with the code or have this tested more empirically than me
(the VM/client node of my test cluster is currently pinning for the
fjords).

When it comes to flushing/evicting we already established that this
triggers based on PG utilization, not a pool wide one.
So for example in a pool with 1024TB capacity (set via target_max_bytes)
and 1024 PGs and a cache_target_dirty_ratio of 0.5 flushing will start
when the first PG reaches 512MB utilization.

However while the documentation states that the least recently objects are
evicted when things hit the cache_target_full_ratio, it is less than clear
(understatement of the year) when flushing is concerned. 
To quote:
"When the cache pool consists of a certain percentage of modified (or
dirty) objects, the cache tiering agent will flush them to the storage pool."

How do we read this?
When hitting 50% (as in the example above) all of the dirty objects will
get flushed? 
That doesn't match what I'm seeing nor would it be a sensible course of
action to unleash such a potentially huge torrent of writes.

If we interpret this as "get the dirty objects below the threshold" (which
is what seems to happen) there are 2 possible courses of action here:

1. Flush dirty object(s) from the PG that has reached the threshold. 
A sensible course of action in terms of reducing I/Os, but it may keep
flushing the same objects over and over again if they happen to be on the
"full" PG.

2. Flush dirty objects from all PGs (most likely in a least recently used
fashion) and stop when we're eventually under the threshold by having
finally hit the "full" PG. 
Results in a lot more IO but will of course create more clean objects
available for eviction if needed.
This is what I think is happening.

So, is there any "least recently used" consideration in effect here,
or is the only way to avoid (pointless) flushes by setting
"cache_min_flush_age" accordingly?

Unlike for flushes above, eviction clearly states that it's going by
"least recently used".
Which in the case of per PG operation would violate that promise, as
people of course expect this to be pool wide.
And if it is indeed pool wide, the same effect as above will happen,
evictions will happen until the "full" PG gets hit, evicting far more than
would have been needed.


Something to maybe consider would be a target value, for example with
"cache_target_full_ratio" at 0.80 and "cache_target_full_ratio_target" at
0.78, evicting things until it reaches the target ratio. 

Lastly, while we have perf counters like "tier_dirty", a gauge for dirty
and clean objects/bytes would be quite useful to me at least.
And clearly the cache tier agent already has those numbers. 
Right now I'm guestimating that most of my cache objects are actually
clean (from VM reboots, only read, never written to), but I have no way
to tell for sure.

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi@xxxxxxx   	Global OnLine Japan/Rakuten Communications
http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux