Re: Ceph-objectstore-tool import failure

[Brady Deetz <bdeetz@xxxxxxxxx>]

It's not simply a zip. I recently went through an incomplete pg incident as well. I'm not sure why your import is failing, but I do know that much. Here's a note in slack from our effort to reverse the export. I'm hoping to explore this a bit more in the next week. 

Data frames appear to have the following format:

ceff DTDT SIZE PAYLOAD

Size is probably in bytes? DTDT is frame type, 8-bits, repeated (so BEGIN=1 is encoded as 0101).

    

File format looks like:

    

Superblock - starts with:   ceff ceff 0200

PG_BEGIN                    ceff 0101 <64 bit little-Endian size>

PG_METADATA                 ceff 0909

    OBJECT_BEGIN            ceff 0303 <64 bit little-Endian size> (represents first object of a file?)

        TYPE_DATA           ceff 0505 (represents an object?)

        (repeat TYPE_DATA frames until file completed)

    TYPE_ATTRS              ceff 0606

    TYPE_OMAP               ceff 0808

    OBJECT_END              ceff 0404

    (repeat above block for all files?)

enum {

    TYPE_NONE = 0,

    TYPE_PG_BEGIN,

    TYPE_PG_END,

    TYPE_OBJECT_BEGIN,

    TYPE_OBJECT_END,

    TYPE_DATA,

    TYPE_ATTRS,

    TYPE_OMAP_HDR,

    TYPE_OMAP,

    TYPE_PG_METADATA,

    TYPE_POOL_BEGIN,

    TYPE_POOL_END,

    END_OF_TYPES, //Keep at the end

};

On Jan 14, 2018 1:27 AM, "Brent Kennedy" <bkennedy@xxxxxxxxxx> wrote:

I was able to bring a server back online for a short time and perform an export of the incomplete PGs I originally posted about last week.  The export showed the files it was exporting and then dropped them all to a PGID.export file.  I then SCP’ed the four PGID.export files to a server where I had an empty OSD weighted to 0.  I stopped that OSD and then tried to import all four PGs.  I then got the following messages for all four I tried:

 

finish_remove_pgs 11.720_head removing 11.720

Importing pgid 11.c13

do_import threw exception error buffer::malformed_input: void object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding

Corrupt input for import

 

 

Command I ran:

ceph-objectstore-tool --op import --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/block --file 11.c13.export

 

The files match the space used by PGs on the disk.  As noted above, I saw it copy the PG to the export file successfully.  Both servers are running Ubuntu 14 with the newest ceph-objectstore-tool installed via the package from here:  http://download.ceph.com/debian-luminous/pool/main/c/ceph/ceph-test_12.2.2-1trusty_amd64.deb  ( cluster is Luminous 12.2.2 .  Its possible the PGs in question are on the jewel version as I wasn’t able to complete the upgrade to luminous on them.

 

Am I missing something?  Can I just copy the files off the failing server via a zip operation locally and then a unzip operation at the destination server?

 

-Brent

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com