Hello, On Wed, 28 Sep 2016 19:36:28 +0200 Sascha Vogt wrote: > Hi Christian, > > Am 28.09.2016 um 16:56 schrieb Christian Balzer: > > 0.94.5 has a well known and documented bug, it doesn't rotate the omap log > > of the OSDs. > > > > Look into "/var/lib/ceph/osd/ceph-xx/current/omap/" of the cache tier and > > most likely discover a huge "LOG" file. > You're right, it was around 200 MB on each OSD (so in total 3,2 GB). > Double restart of the OSDs fixed that. Can this log file keep 0-byte > files alive? In other words could this LOG file be the reason for the > missing 840 GB - or did you think that the log itself might be in the GB > range? > The later, if not watched long enough. But yeah, not explaining the amounts you were mentioning. I don't think the LOG is keeping the 0-byte files alive, though. In general these are objects that have been evicted from the cache and if it's very busy you will wind up with each object that's on the backing pool also being present (if just as 0-byte file) in your cache tier. Similar to objects that get created on the cache-tier (writes) and have not been flushed, they will have 0-byte file on the backing pool. So that is going to eat up space in a fashion. In your particular case, I'd expect objects that are deleted to be gone, maybe with some delay. Can you check/verify that the deleted objects are actually gone on the backing pool? > Anyway, already thanks for the hint about the log file. We'll keep an > eye on that one and try to upgrade to Hammer soon! > Well you're already on Hammer. ^o^ Just don't upgrade to 0.94.6, whatever you do (lethal cache tier bug). If you don't have too many OSDs (see the various threads here), upgrading to 0.94.9 or a to be released .10 which addresses the encoding storms should be fine. At this point in time I think Jewel still has too many rough edges, but that's me. Take note (search the ML archives) that Jewel massively changes the cache tiering behavior (not promoting things as readily as Hammer), so make sure you don't get surprised there. Christian > Greetings > -Sascha- > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |