On 07/14/2013 04:27 AM, James Harper wrote:
My cluster is in HEALTH_WARN state because one of my monitors has low disk space on /var/lib/ceph. Looking into this in more detail, there are a bunch of .sst files dating back to Jul 7, and then a lot more at Jun 30 and older.
Since cuttlefish's release, there were a bunch of issues with leveldb not compacting (i.e., garbage collecting removed key ranges). This made the store grow when adding new keys to the store as part of the monitor's normal workload.
As James Harper said, one of the approaches to take care of this has been restarting the monitors on cuttlefish versions up to 0.61.3. On 0.61.4 we added a couple of patches that ought to make that behavior go away. If you're not running 0.61.4 you should upgrade, as there were other important bugs that were fixed besides the leveldb compaction issues -- and then let us know how it went :-)
I'm thinking that these older files are just ones that mon has failed to clean up over time... I can't imagine that mon needs 3GB of historic data.
I've seen monitors keeping several GBs of data. Basically, the monitors will keep osdmaps. These are only removed when the osds are healthy. For clusters with large amounts of OSDs there may be a chance that a monitor will keep a lot of osdmaps (hence a lot of data, even GB range). However, this doesn't seem to be the case: looks like you're being bit by leveldb's compaction issues.
Is there a 'clean up' command, or can I just delete them manually?
You should never delete the files manually. They are managed entirely by leveldb and if you mess with them it's very likely your mon store will end up corrupted.
You may try 'ceph -m IP:PORT compact', with IP and PORT being the monitor you want to compact IP and PORT. Also, you could restart your monitor with '--compact' and it should have pretty much the same effect.
Or you could upgrade to 0.61.4 if you haven't done so :-) -Joao
Thanks James _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-- Joao Eduardo Luis Software Engineer | http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
