Hi Don, I experienced the same thing a couple of days ago on Hammer. On investigation the cache mode wasn't set to writeback even though I'm sure it accepted the command successfully when I set the pool up. Could you reapply the cache mode writeback command and see if that makes a difference? Nick > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Don Doerner > Sent: 30 April 2015 17:57 > To: ceph-users@xxxxxxxxxxxxxx > Subject: RHEL7/HAMMER cache tier doesn't flush or evict? > Sensitivity: Personal > > All, > > Synopsis: I can't get cache tiering to work in HAMMER on RHEL7. > > Process: > 1. Fresh install of HAMMER on RHEL7 went well. > 2. Crush map adapted to provide two "root" level resources > a. "ctstorage", to use as a cache tier based on very high-performance, high > IOPS SSD (intrinsic journal). 2 OSDs. > b. "ecstorage", to use as an erasure-coded poolbased on low-performance, > cost effective storage (extrinsic journal). 12 OSDs. > 3. Established a pool "ctpool", 32 PGs on ctstorage (pool size = min_size = 1). > Ran a quick RADOS bench test, all worked as expected. Cleaned up. > 4. Established a pool "ecpool", 256 PGs on ecstorage (5+3 profile). Ran a > quick RADOS bench test, all worked as expected. Cleaned up. > 5. Ensured that both pools were empty (i.e., "rados ls" shows no objects) > 6. Put the cache tier on the erasure coded storage (one Bloom hit set, > interval 900 seconds), set up the overlay. Used defaults for flushing and > eviction. No errors. > 7. Started a 3600-second write test to ecpool. > > Objects piled up in ctpool (as expected) - went past the 40% mark (as > expected), then past the 80% mark (unexpected), then ran into the wall > (95% full - very unexpected). Using "rados df", I can see that the cache tier is > full (duh!) but not one single object lives in the ecpool. Nothing was ever > flushed, nothing was ever evicted. Thought I might be misreading that, so I > went back to SAR data that I captured during the test: the SSDs were the only > [ceph] devices that sustained I/O. > > I based this experiment on another (much more successful) experiment that > I performed using GIANT (.1) on RHEL7 a couple of weeks ago (wherein I > used RAM as a cache tier); that all worked. It seems there are at least three > possibilities. > . I forgot a critical step this time around. > . The steps needed for a cache tier in HAMMER are different than the steps > needed in GIANT (and different than the documentation online). > . There is a problem with HAMMER in the area of cache tier. > > Has anyone successfully deployed cache-tiering in HAMMER? Did you have > to do anything unusual? Do you see any steps that I missed? > > Regards, > > -don- > > ________________________________________ > The information contained in this transmission may be confidential. Any > disclosure, copying, or further distribution of confidential information is not > permitted unless such privilege is explicitly granted in writing by Quantum. > Quantum reserves the right to have electronic communications, including > email and attachments, sent across its networks filtered through anti virus > and spam software programs and retain such messages in order to comply > with applicable data security and retention requirements. Quantum is not > responsible for the proper and complete transmission of the substance of > this communication or for any delay in its receipt. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |