22:16 ccallegar: good grief...talk about a handful of sand in your eye! I've been chasing down a "leveldb: Compaction error: Corruption: bad entry in block " in mons logs...
22:17 ccallegar: I ran a python leveldb.repair() and restarted osd's and mons and my cluster crashed and burned
22:18 ccallegar: a couple files ended up in leveldb lost dirs. The path is different if it's a mons or osd
22:19 ccallegar: for mons logs showed a MANIFEST file missing. I moved the file that landed in lost back to normal position, chown'd ceph:ceph, restarted mons and mons came back online!
22:21 ccallegar: osd logs showed a sst file missing. looks like leveldb.repair() does the needful but names the new file a .ldb. I renamed the file, chown'd ceph:ceph, restarted osd and they came back online!
22:17 ccallegar: I ran a python leveldb.repair() and restarted osd's and mons and my cluster crashed and burned
22:18 ccallegar: a couple files ended up in leveldb lost dirs. The path is different if it's a mons or osd
22:19 ccallegar: for mons logs showed a MANIFEST file missing. I moved the file that landed in lost back to normal position, chown'd ceph:ceph, restarted mons and mons came back online!
22:21 ccallegar: osd logs showed a sst file missing. looks like leveldb.repair() does the needful but names the new file a .ldb. I renamed the file, chown'd ceph:ceph, restarted osd and they came back online!
leveldb corruption log entries have gone away and my cluster is recovering it's way to happiness.
Hopefully this helps someone else out
Thanks,
/Chris
On Tue, Jun 27, 2017 at 6:39 PM, Mazzystr <mazzystr@xxxxxxxxx> wrote:
Hi Ceph Users,I've been chasing down some levelDB corruption messages in my mons logs. I ran a python leveldb repair on mon and odd leveldbs. The job caused a files to disappear and a log file to appear in lost directory. Mon and osd's refuse to boot.Ceph version is kraken 11.02.There's not a whole lot of info on the internet regarding this. Anyone have any ideas on how to recover the mess?Thanks,/Chris C
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
