Re: Major ceph disaster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On Sat, May 25, 2019 at 7:45 PM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:


On Fri, May 24, 2019 at 5:22 PM Kevin Flöh <kevin.floeh@xxxxxxx> wrote:

ok this just gives me:

error getting xattr ec31/10004dfce92.00000000/parent: (2) No such file or directory

Try to run it on the replicated main data pool which contains an empty object for each file, not sure where the xattr is stored in a multi-pool setup.

Also, you probably didn't lose all the chunks of the erasure coded data. Check the list_missing output to see which chunks are still there and where they are.
You can export the chunks that you still have using ceph-objectstore-tool. The first 3 chunks will be the data of the object, you might be able to tell if that file is import for you.


Paul
 



--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
 

Does this mean that the lost object isn't even a file that appears in the ceph directory. Maybe a leftover of a file that has not been deleted properly? It wouldn't be an issue to mark the object as lost in that case.

On 24.05.19 5:08 nachm., Robert LeBlanc wrote:
You need to use the first stripe of the object as that is the only one with the metadata. 

Try "rados -p ec31 getxattr 10004dfce92.00000000 parent" instead.

Robert LeBlanc

Sent from a mobile device, please excuse any typos.

On Fri, May 24, 2019, 4:42 AM Kevin Flöh <kevin.floeh@xxxxxxx> wrote:

Hi,

we already tried "rados -p ec31 getxattr 10004dfce92.0000003d parent" but this is just hanging forever if we are looking for unfound objects. It works fine for all other objects.

We also tried scanning the ceph directory with find -inum 1099593404050 (decimal of 10004dfce92) and found nothing. This is also working for non unfound objects.

Is there another way to find the corresponding file?

On 24.05.19 11:12 vorm., Burkhard Linke wrote:

Hi,

On 5/24/19 9:48 AM, Kevin Flöh wrote:

We got the object ids of the missing objects with ceph pg 1.24c list_missing:

{
    "offset": {
        "oid": "",
        "key": "",
        "snapid": 0,
        "hash": 0,
        "max": 0,
        "pool": -9223372036854775808,
        "namespace": ""
    },
    "num_missing": 1,
    "num_unfound": 1,
    "objects": [
        {
            "oid": {
                "oid": "10004dfce92.0000003d",
                "key": "",
                "snapid": -2,
                "hash": 90219084,
                "max": 0,
                "pool": 1,
                "namespace": ""
            },
            "need": "46950'195355",
            "have": "0'0",
            "flags": "none",
            "locations": [
                "36(3)",
                "61(2)"
            ]
        }
    ],
    "more": false
}

we want to give up those objects with:

ceph pg 1.24c mark_unfound_lost revert

But first we would like to know which file(s) is affected. Is there a way to map the object id to the corresponding file?


The object name is composed of the file inode id and the chunk within the file. The first chunk has some metadata you can use to retrieve the filename. See the 'CephFS object mapping' thread on the mailing list for more information.


Regards,

Burkhard



_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux