Re: Ceph PGs stuck creating after running force_create_pg

Goncalo Borges <goncalo@xxxxxxxxxxxxxxxxxxx> · Thu, 15 Oct 2015 18:32:28 +1100



    Hi James...

    
    I am assuming that you have properly removed the dead OSDs from the
    crush map.

    
    I've tested a scenario like the one you described and realized that
    pgs will never leave the creating state until you restart all OSDs.

    
    Have you done that?

    
    Cheers

    Goncalo

    
    On 10/15/2015 01:54 AM, James Green
      wrote:

    
        Hello,
         
        We recently had 2 nodes go down in our ceph
          cluster, one was repaired and the other had all 12 osds
          destroyed when it went down. We brought everything back
          online, there were several PGs that were showing as
          down+peering as well as down. After marking the failed OSDs as
          lost and removing them from the cluster we now have around 90
          PGs that are showing as incomplete. At this point we just want
          to get the cluster back up and in a healthy state. I tried
          recreating the PGs using force_create_pg and now they are all
          stuck in creating. 
         
        PG dump shows 90 pgs all with the same
          output 
        2.182   0       0       0       0      
          0       0       0       0       creating        2015-10-14
          10:31:28.832527      0'0     0:0     []      -1      []     
          -1      0'0     0.000000        0'0     0.000000
         
        When I ran pg query on one of the groups I
          noticed under "down_osds_we_would_probe" one of the failed
          OSDs was listed. I already removed the OSD from the cluster,
          trying to mark it lost says the OSD does not exist.
          
         
        Here is my crushmap 
            http://pastebin.com/raw.php?i=vyk9vMT1 
         
        Why are the PGs trying to query osds that
          have been lost and removed from the cluster?
          
         
      _______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

    
    -- 
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW  2006
T: +61 2 93511937

  
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com