Re: ceph-objectstore-tool crash when trying to recover pg from OSD

Igor Fedotov <ifedotov@xxxxxxx> · Fri, 8 Nov 2019 02:54:05 +0300



    Hi Eugene,
    this looks like https://tracker.ceph.com/issues/42223
      indeed.
    Would you please find the first crash for these OSDs and share
      corresponding logs in the ticket.
    

    Unfortunately I don't know reliable enough ways to recover OSD
      after such a failure. If they exist at all... :(
    I've been told offline by Rafal Wadolowski that
      ceph-kvstore-tool's destructive-repair command helped in 1 of 2
      attempts. But I would strictly  advise to refrain from this
      command usage for now unless you're  absolutely immune to data
      loss. It might make things even worse....

    
    Thanks,
    Igor

    
    On 11/7/2019 11:56 AM, Eugene de Beste
      wrote:

    
          Hi, does anyone have any feedback for me regarding this?
        
        
        Here's the log I get when trying to restart the OSD via
          systemctl: https://pastebin.com/tshuqsLP

        
            On Mon, 4 Nov 2019 at 12:42, Eugene de Beste <eugene@xxxxxxxxxxx>
              wrote:
          
          
            Hi everyone
            

            I have a cluster that was initially set up with bad
              defaults in Luminous. After upgrading to Nautilus I've had
              a few OSDs crash on me, due to errors seemingly related to
              https://tracker.ceph.com/issues/42223
              and https://tracker.ceph.com/issues/22678.
            

            One of my pools have been running in min_size 1 (yes, I
              know) and I am not stuck with incomplete pgs due to
              aforementioned OSD crash.
            

            When trying to use the ceph-objectstore-tool to get the
              pgs out of the OSD, I'm running into the same issue as
              trying to start the OSD, which is the crashes.
              ceph-objectstore-tool core dumps and I can't retrieve the
              pg.
            

            Does anyone have any input on this? I would like to be
              able to retrieve that data if possible.
            

            Here's the log for ceph-objectstore-tool --debug
              --data-path /var/lib/ceph/osd/ceph-22
              --skip-journal-replay --skip-mount-omap --op info --pgid
              2.9f  -- https://pastebin.com/9aGtAfSv
            

            Regards and thanks,
            Eugene
          
        
      _______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

    
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com