Hello, On Thu, 29 Sep 2016 20:15:12 +0200 Sascha Vogt wrote: > Hi Burkhard, > > On 29/09/16 15:08, Burkhard Linke wrote: > > AFAIK evicting an object also flushes it to the backing storage, so > > evicting a live object should be ok. It will be promoted again at the > > next access (or whatever triggers promotion in the caching mechanism). > >> > >> For the dead 0-byte files: Should I open a bug report? > > Not sure whether this is a bug at all. The objects should be evicted and > > removed if the cache pool hits the max object thresholds. > d'oh, Ceph and it's hidden gems ;) That was it. That's what I alluding to when I wrote "maybe with some delay". >Yes, we have currently > no hard object limit (target_max_objects) as we have target_max_bytes > set and thought that would be enough. After setting target_max_objects > (even to a ridiculous high number, I used 200 millions, so double the > amount we have) and Ceph immediately started dropping objects (and > blocking all client IO :( ) > Please refer to this page for the reminder: http://docs.ceph.com/docs/master/rados/operations/cache-tiering/ So, firstly, what are your ratios (dirty, full) set to? If it's at the defaults of 0.4 and 0.6 and you REALLY have only 100 million objects, it should have started to flush stuff (which is likely a NOOP with these leftovers) and not evict stuff. What does "ceph df detail" tell you? Are you sure the blocking of client I/O is due to the object removal and your OSDs being too busy and not actually because Ceph thinks that the cache is full (object wise)? As in: "Note All client requests will be blocked only when target_max_bytes or target_max_objects reached" > Is this behavior documented somewhere? Not that I'm aware of. OTOH, I'd expect even those 0-byte files/objects to be eventually the subject of removal when the space/size limits are reached and they are eligible (old enough). If that is NOT the case, that this is both a bug and at the very least needs to be put into the documentation. >From the cache tiering doc it > looked like you either set target_max_bytes OR target_max_objects and > not both (although I always wondered what sense does it make to talk > about objects on a cache layer, as it's nature is that it is space bound > and it is less than the backing pool. I even wondered why > target_max_bytes is even necessary, as Ceph knows how much space is > available. As it says on that page, it doesn't. Ceph is notoriously bad (due to the potential complexity of setups) to figure out what space is actually available and used. This is isn't helped by cache-tiering basing things on PGs not pools or OSDs or anything else that would help to make sizing guesses. See my old "Cache tier operation clarifications" thread here. Christian > I mean optionally restricting it further is ok, in case you > want to have two Cache pools on the same set of fast disks / SSds, but > IMHO it could be optionally and in case of just one pool use whats there) > > Anyway, thanks a lot for the help. We will see how we can get some > downtime in order to set a limit and cleanup the backlog of stale > objects from the cache. > > Greetings > -Sascha- > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |