leveldb on OSD with missing file after hard boot

Gerd Jakobovitsch <gerd@xxxxxxxxxxxxx> · Wed, 27 Jan 2016 14:36:42 -0200

    Hello
        all,

        I had a hard reset on a ceph node, and one of the OSDs is not
        starting due to leveldb error. At that moment, the node was
        trying to start up, but there was no actual writing of new data:

        2016-01-27 12:00:37.068431 7f367f654880  0 ceph version 0.94.5
        (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd,
        pid 24734

        2016-01-27 12:00:37.115800 7f367f654880  0
        filestore(/var/lib/ceph/osd/ceph-26) backend xfs (magic
        0x58465342)

        2016-01-27 12:00:37.133031 7f367f654880  0
        genericfilestorebackend(/var/lib/ceph/osd/ceph-26)
        detect_features: FIEMAP ioctl is supported and appears to work

        2016-01-27 12:00:37.133042 7f367f654880  0
        genericfilestorebackend(/var/lib/ceph/osd/ceph-26)
        detect_features: FIEMAP ioctl is disabled via 'filestore fiemap'
        config option

        2016-01-27 12:00:37.136538 7f367f654880  0
        genericfilestorebackend(/var/lib/ceph/osd/ceph-26)
        detect_features: syncfs(2) syscall fully supported (by glibc and
        kernel)

        2016-01-27 12:00:37.137584 7f367f654880  0
        xfsfilestorebackend(/var/lib/ceph/osd/ceph-26) detect_feature:
        extsize is supported and kernel 3.10.0-123.el7.x86_64 >= 3.5

        2016-01-27 12:00:37.176226 7f367f654880 -1
        filestore(/var/lib/ceph/osd/ceph-26) Error initializing leveldb
        : Corruption: 1 missing files; e.g.:
        /var/lib/ceph/osd/ceph-26/current/omap/075074.sst

        2016-01-27 12:00:37.176286 7f367f654880 -1 osd.26 0 OSD:init:
        unable to mount object store

        2016-01-27 12:00:37.176315 7f367f654880 -1  ** ERROR: osd init
        failed: (1) Operation not permitted

        The file 075074.sst is missing indeed.

        Since I was not able to restart the OSD, and I could not find
        information on recovering the leveldb, I marked the OSD as lost,
        but then I got 3 incomplete OSDs. I tried to follow the recovery
        howto at https://ceph.com/community/incomplete-pgs-oh-my/, but
        it stepped on the same error on leveldb, having the same
        dependency.

        Is there any means to recover from this situation? To check and
        recover leveldb as good as possible? Or, alternatively, to get
        rid of the incomplete status, even with the penalty of losing
        some objects?

        Regards.

--

As informações contidas nesta mensagem são CONFIDENCIAIS, protegidas pelo sigilo legal e por direitos autorais. A divulgação, distribuição, reprodução ou qualquer forma de utilização do teor deste documento depende de autorização do emissor, sujeitando-se o infrator às sanções legais. Caso esta comunicação tenha sido recebida por engano, favor avisar imediatamente, respondendo esta mensagem.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com