-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 My experience with Hammer is showing that setting the pool to forward mode is not evicting objects, nor do I think it is flushing objects. We have had our pool in forward mode for weeks now and we still have almost the same amount of I/O to it. There has been a slight shift between SSD and HDD, but I think that is because some objects have cooled off and others have been newly accessed. You may have better luck adjusting the ratios, but we see that there is a big hit to our cluster to do that. We usually do 1% every minute or two to help reduce the impact of evicting the data (we usually drop the cache full ratio 10% or so to evict some objects and we then toggle the cache mode between writeback and forward periodically to warm up the cache. Setting it to writeback will promote so many objects at once that it severely impact our cluster. There is also a limit that we reach at about 10K IOPs when in writeback where with forward I've seen spikes to 64K IOPs. So we turn on writeback for 30-60 seconds (or until the blocked I/O is too much for us to handle), then set it to forward for 60-120 second, rinse and repeat until the impact of writeback isn't so bad, then set it back to forward for a couple more weeks). Needless to say, cache tiering could use some more love. If I get some time, I'd like to try and help that section of code, but I have a couple other more pressing issues I'd like to track down first. -----BEGIN PGP SIGNATURE----- Version: Mailvelope v1.3.4 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWspPSCRDmVDuy+mK58QAAsQgP/15YrzV+BRt+CGnzZL/Q w6PwnSdw4HBJT4OEqdg+kStCP+SqUSVCiJcdeHo5Sm40smEWVYRim3jsHBSg Z4Woa31XsjYbEw3HCxIoI93OPhaKszOhvktKZxu1iSnyMDDJIYMARlYIjbfc ToCOC/IVe2MMAEtVq+J2fm/NQy6VDGbaUuYcNtkIF41j7vKoNoE3h5qi+L0K cVwUhVTcuSNDuiuJOoduM/vSH6nJzmCnypH1BDTcEOYpvmbXWJ0iTdej2Oa1 gVvV7SOcu4PkjzL9MmJB2Cjiiy/zWjUTfN01nBvIatwOjF7AE8vq2pLD9FIs TxmzE4UZgjwJNbkDVQsgHPCeUlEll+t3QKbokpEkQDQgvIOs6NCbj0KYpuhC DWtQCbgYsniT+Md1vWFMgqs0a45ulGxEKUWiUOEXgTJLHH+dbrW32MZEl1Gd yTKyzFarbae6tbAmaMPC8l9vaj15t7bAB0KOokMqZied7EcM1ZoFVqKRahrm 73mIeHiDUwZ8gi+BHKX7OwqKt3VZJYf/+rNJx+g4kp5WN0FEkUMoqF75qO4p 62+PuQIwh6jUpB4cDsbEJd78UGbCptJBojmsNVogU+xiSXTKQmEduP0HqQfG JhTLg3Un2C4/MSGbhRI26csFCzEi66iRXQWdfCITP4Um70KO6dE2C1MAveYg hJ7b =CaRF -----END PGP SIGNATURE----- ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Feb 3, 2016 at 10:01 AM, Nick Fisk <nick@xxxxxxxxxx> wrote: > I think this would be better to be done outside of Ceph. It should be quite simple for whatever monitoring software you are using to pick up the disk failure to set the target_dirty_ratio to a very low value or change the actual caching mode. > > Doing it in Ceph would be complicated as you are then asking Ceph to decide when you are in an at risk scenario, ie would you want it to flush your cache after a quick service reload or node reboot? > >> -----Original Message----- >> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of >> Mihai Gheorghe >> Sent: 03 February 2016 16:57 >> To: ceph-users <ceph-users@xxxxxxxxxxxxxx>; ceph-users <ceph- >> users@xxxxxxxx> >> Subject: Set cache tier pool forward state automatically! >> >> Hi, >> >> Is there a built in setting in ceph that would set the cache pool from >> writeback to forward state automatically in case of an OSD fail from the pool? >> >> Let;s say the size of the cache pool is 2. If an OSD fails ceph blocks write to >> the pool, making the VM that use this pool to be unaccesable. But an earlier >> copy of the data is present on the cold storage pool prior to the last cache >> flush. >> >> In this case, is it possible that when an OSD fails, the data on the cache pool >> to be flushed onto the cold storage pool and set the forward flag >> automatically on the cache pool? So that the VM can resume write to the >> block device as soon as the cache is flushed from the pool and read/write >> directly from the cold storage pool untill manual intervention on the cache >> pool is done to fix it and set it back to writeback? >> >> This way we can get away with a pool size of 2 without worrying for too much >> downtime! >> >> Hope i was explicit enough! > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |