Re: The journey to CephFS metadata pool’s recovery

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Have you tried to hexdump the actual NVMEs devices instead of the testdisk images? 

`hexdump -C -n 4096 /dev/nvmexxx` should show LVM LABELONE with PV UUID 

and 

`hexdump -C -n 4096 /dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b` should show bluestore metadata AKA 'label' that you can get with `ceph-bluestore-tool show-label` 

If nothing shows up than you may need to rewrite the label as described here [1] by Igor. Thing is, you'd have to make sure you're dealing with the right disk, meaning both NVMEs were not swapped. 

To help you with that part, you could run `strings /dev/nvmexxx | less` and try to link the VG informations with OSD config files if you're able to. 

Hope that helps. 

Regards, 
Frédéric. 

[1] https://www.spinics.net/lists/ceph-users/msg81813.html 

----- Le 3 Sep 24, à 11:56, Marco Faggian <m@xxxxxxxxxxxxxxxx> a écrit : 

> Hi Frédéric,
> Thanks a lot for the pointers!

> So, using testdisk I’ve created images of both the LVs. I’ve looked at the
> hexdump and it’s filled with 0x00 until 00a00000.
> Then for curiosity I’ve compared them and they’re identical until byte 12726273.

> Also unfortunately the issue is that ceph-bluestore-tool show-label, like
> dumpe2fs -h /dev/ceph-3.. are both erroring out in the same way:

> unable to read label for
> /dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b:
> 2024-09-03T11:27:22.938+0200 7f2d89f19a00 -1
> bluestore(/dev/ceph-454751de-44ab-4aa6-b3ae-50abc22250b3/osd-block-b7745d63-0bf8-4ba4-9274-e034f1c15d7b)
> _read_bdev_label unable to decode label at offset 102: void
> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)
> decode past end of struct encoding: Malformed input [buffer:3]

> (2) No such file or directory

> Unfortunately the thread on the tracker doesn’t seem to point to a solution,
> even though the issue seems to be identical.

> The hexdump explains why it’s not finding the label, might it be that the LV is
> not correctly mapped?

> Basically here the question is: is there a way to recover the data of an OSD in
> an LV, if it was ceph osd purge before the cluster had a chance to replicate it
> (after ceph osd out )?

> Thanks for your time!
> fm

>> On 3 Sep 2024, at 10:35, Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx> wrote:

>> Hi Marco,

>> Have you checked the output of:

>> dd if=/dev/ceph-xxxxxxx/osd-block-xxxxxxxxx of=/tmp/foo bs=4K count=2
>> hexdump -C /tmp/foo

>> and:

>> /usr/bin/ceph-bluestore-tool show-label --log-level=30 --dev /dev/nvmexxx -l
>> /var/log/ceph/ceph-volume.log

>> to see if it's aligned with OSD's metadata.

>> You may also want to check this discussion [1] and this tracker [2] for useful
>> commands.

>> Regards,
>> Frédéric

>> [1] https://marc.info/?l=ceph-users&m=171395775626007&w=2
>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1755956

>> ----- Le 2 Sep 24, à 11:26, m@xxxxxxxxxxxxxxxx a écrit :

>>>> FYI: Also posted in L1Techs forum:
>>>> https://forum.level1techs.com/t/recover-bluestore-osd-in-ceph-cluster/215715.

>>> ## The epic intro
>>> Through self-inflicted pain, I’m writing here to ask for volunteers in the
>>> journey of recovering the lost partitions housing CephFS metadata pool.

>>> ## The setup
>>> 1 proxmox host (I know)
>>> 1 replication rule only for NVMes (2x OSD)
>>> 1 replication rule only for HDDs (8x OSD)
>>> Each with failure domain to osd.
>>> Each OSD configured to use bluestore backend in an LVM.
>>> No backup (I know, I know).

>>> ## The cause (me)
>>> Long story short: I needed the PCIe lanes and decided to remove the two NVMEs
>>> that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to
>>> remove the two OSDs (out and destroy).
>>> This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one,
>>> so the cluster never moved the PGs stored in the NVMes, to the HDDs.

>>> ## What I’ve done untill now
>>> 1. Re-seated the NVMes to their original place.
>>> 2. Found out that the LVM didn’t have the OSD’s labels applied
>>> 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity
>>> that thought that archiving LVM config was a good thing, it payed back)
>>> 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode
>>> label at offset 102 in the LVM for that ODS.

>>> ## Wishes
>>> 1. Does anyone know a way to recover what I feel is a lost partition, given that
>>> the “file system” is ceph’s bluestore?
>>> 2. Is there a way to know, if it is, how the partition has been nuked? And
>>> possibly find a way to reverse that process.

>>> ## Closing statement
>>> Eternal reminder: If you don’t want to lose it, back it up.
>>> Thanks for your time, to the kind souls that are willing to die on this hill
>>> with me, or come up victorious!
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux