Is there anything you can do with a USB key/NFS mount? Ie copy leveldb on to it, remount in proper location, compact and then copy back to primary disk? > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Wido den Hollander > Sent: 20 January 2016 15:15 > To: ceph-users@xxxxxxxxxxxxxx > Subject: Ceph monitors 100% full filesystem, refusing start > > Hello, > > I have an issue with a (not in production!) Ceph cluster which I'm trying to > resolve. > > On Friday the network links between the racks failed and this caused all > monitors to loose connection. > > Their leveldb stores kept growing and they are currently 100% full. They all > have a few hunderd MB left. > > Starting the 'compact on start' doesn't work since the FS is 100% > full.error: monitor data filesystem reached concerning levels of available > storage space (available: 0% 238 MB) you may adjust 'mon data avail crit' to a > lower value to make this go away (default: 0%) > > On of the 5 monitors is now running but that's not enough. > > Any ideas how to compact this leveldb? I can't free up any more space right > now on these systems. Getting bigger disks in is also going to take a lot of > time. > > Any tools outside the monitors to use here? > > Keep in mind, this is a pre-production cluster. We would like to keep the > cluster and fix this as a good exercise of stuff which could go wrong. > Dangerous tools are allowed! > > -- > Wido den Hollander > 42on B.V. > Ceph trainer and consultant > > Phone: +31 (0)20 700 9902 > Skype: contact42on > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |