Re: PG's stuck inactive, stuck unclean, incomplete, imports cause osd segfaults - Hire a consultant

Richard Arends <cephmailinglist@xxxxxxxxx> · Thu, 3 Mar 2016 18:38:56 +0100

      On 03/03/2016 06:12 PM, Philip Hempel
        wrote:

        Philip,

        I forgot to CC the list, now i did...

        To export the data I used ceph-objectstore-tool
          to do this using the export command.

          I am trying to repair a cluster that has 74 pgs that are down,
          I have seen that the pgs in question are presently with 0 data
          on the OSD.

              Can you show the output of query of one pg? "ceph pg
              <pg_id> query"

               I
                have exported data from OSD's that were pulled when the
                client had thought the disk were bad.

              How did you export the data ?

      Okay, from the pg query output i get the following:

      $ egrep 'peer_info|\"peer\"|\"num_bytes\"' ceph-query-3.50f 

                      "num_bytes": 0,

          "peer_info": [

                  "peer": "5",

                          "num_bytes": 0,

                  "peer": "9",

                          "num_bytes": 0,

                  "peer": "10",

                          "num_bytes": 0,

                  "peer": "17",

                          "num_bytes": 0,

                  "peer": "19",

                          "num_bytes": 0,

                  "peer": "23",

                          "num_bytes": 897560576,

                  "peer": "24",

                          "num_bytes": 0,

                  "peer": "39",

                          "num_bytes": 0,

                  "peer": "43",

                          "num_bytes": 0,

                  "peer": "44",

                          "num_bytes": 0,

                  "peer": "48",

                          "num_bytes": 0,

      So osd.23 looks like the only that has data and the acting OSDs
      does not have any.  On which OSD did you import the data? Since
      osd.34 and osd.43 are acting for this PG, maybe you can try to
      import the data (also) on those OSDs. Before importing, set the
      cluster to noout and stop the OSD daemon where you will import the
      data. I think you als must remove the present PG info on the disk,
      for example "rm /var/lib/ceph/osd/ceph-34/current/3.50f_head", but
      check (and double check) that you only remove that directory when
      it is empty....

      -- 
Regards,

Richard.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com