Re: ceph-objectstore-tool crash when trying to recover pg from OSD

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Eugene,

this looks like https://tracker.ceph.com/issues/42223 indeed.

Would you please find the first crash for these OSDs and share corresponding logs in the ticket.


Unfortunately I don't know reliable enough ways to recover OSD after such a failure. If they exist at all... :(

I've been told offline by Rafal Wadolowski that ceph-kvstore-tool's destructive-repair command helped in 1 of 2 attempts. But I would strictly  advise to refrain from this command usage for now unless you're  absolutely immune to data loss. It might make things even worse....

Thanks,

Igor

On 11/7/2019 11:56 AM, Eugene de Beste wrote:
Hi, does anyone have any feedback for me regarding this?

Here's the log I get when trying to restart the OSD via systemctl: https://pastebin.com/tshuqsLP
On Mon, 4 Nov 2019 at 12:42, Eugene de Beste <eugene@xxxxxxxxxxx> wrote:
Hi everyone

I have a cluster that was initially set up with bad defaults in Luminous. After upgrading to Nautilus I've had a few OSDs crash on me, due to errors seemingly related to https://tracker.ceph.com/issues/42223 and https://tracker.ceph.com/issues/22678.

One of my pools have been running in min_size 1 (yes, I know) and I am not stuck with incomplete pgs due to aforementioned OSD crash.

When trying to use the ceph-objectstore-tool to get the pgs out of the OSD, I'm running into the same issue as trying to start the OSD, which is the crashes. ceph-objectstore-tool core dumps and I can't retrieve the pg.

Does anyone have any input on this? I would like to be able to retrieve that data if possible.

Here's the log for ceph-objectstore-tool --debug --data-path /var/lib/ceph/osd/ceph-22 --skip-journal-replay --skip-mount-omap --op info --pgid 2.9f  -- https://pastebin.com/9aGtAfSv

Regards and thanks,
Eugene

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux