Hey Dan, This is on Hammer 0.94.5. osd.52 was always on a problematic machine and when this happened had less data on its local disk than the other OSDs. I've tried adapting that blog post's solution to this situation to no avail. I've tried things like looking at all probing OSDs in the query output and importing the data from one copy to all of them to get it to be consistent. One of the major red flags here was that when I looked at the original acting set's disks I found each OSD had a different amount of data for the same PG, there is at least one PG here where 52 (the primary for all four) actually had about 1GB (~27%) less data, everything has just been really inconsistent. Here's hoping Cunningham will come to the rescue. Cheers, George ________________________________________ From: Dan van der Ster [dan@xxxxxxxxxxxxxx] Sent: 11 May 2016 17:28 To: Vasilakakos, George (STFC,RAL,SC) Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: Incomplete PGs, how do I get them back without data loss? Hi George, Which version of Ceph is this? I've never had incompete pgs stuck like this before. AFAIK it means that osd.52 would need to be brought up before you can restore those PGs. Perhaps you'll need ceph-objectstore-tool to help dump osd.52 and bring up its data elsewhere. A quick check on this list pointed to https://ceph.com/community/incomplete-pgs-oh-my/ -- did you try that? Or perhaps I'm spewing enough nonsense here that Cunningham's Law will bring you the solution. Cheers, Dan On Thu, May 5, 2016 at 8:21 PM, <george.vasilakakos@xxxxxxxxxx> wrote: > Hi folks, > > I've got a serious issue with a Ceph cluster that's used for RBD. > > There are 4 PGs stuck in an incomplete state and I'm trying to repair this problem to no avail. > > Here's ceph status: > health HEALTH_WARN > 4 pgs incomplete > 4 pgs stuck inactive > 4 pgs stuck unclean > 100 requests are blocked > 32 sec > monmap e13: 3 mons at ... > election epoch 2084, quorum 0,1,2 mon4,mon5,mon3 > osdmap e154083: 203 osds: 197 up, 197 in > pgmap v37369382: 9856 pgs, 5 pools, 20932 GB data, 22321 kobjects > 64871 GB used, 653 TB / 716 TB avail > 9851 active+clean > 4 incomplete > 1 active+clean+scrubbing > > The 4 PGs all have the same primary OSD, which is on a host that had its OSDs turned off as it was quite flaky. > > 1.1bdb incomplete [52,100,130] 52 [52,100,130] 52 > 1.5c2 incomplete [52,191,109] 52 [52,191,109] 52 > 1.f98 incomplete [52,92,37] 52 [52,92,37] 52 > 1.11dc incomplete [52,176,12] 52 [52,176,12] 52 > > One thing that strikes me as odd is that once osd.52 is taken out, these sets change completely. > The situation currently is that, for each of these PGs, the three OSDs have different amounts of data. > They all have similar but different amounts, with osd.52 having the smallest amount (not by too much though) in each case. > > Querying those PGs doesn't return a response after a few minutes, manually triggering scrubs or repairs on them does nothing. > I've lowered the min_size from 2 to 1 but I'm not seeing any activity to fix this. > > Is there something that can be done to recover without losing that data (it means each VM has a 75% chance of being destroyed)? > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |