Re: Unclean PGs in active+degrared or active+remapped

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I'm by no means an expert, but from what I understand you do need to stick to numbering from zero if you want things to work out in the long term.  Is there a chance that the cluster didn't finish bringing things back up to full replication before osd's were removed?  

If I were moving from 0,1 to 2,3 I'd bring both 2 and 3 up, set the weight of 0,1 to zero and let all of the pg's get active+clean again then remove 0,1.  Doing your swap I might bring up 2 under rack az2, set 1 to weight 0, stop 1 after getting active+clean and remake what is now 3 as 1 and bring it back in as 1 with full weight, then finally drop 2 to weight zero and remove after active+clean.  I'd follow on doing a similar shuffle for the now inactive former osd 1 the current osd 0 and the future osd 0 which was osd 2.  Clear as mud?


On Jul 19, 2013, at 7:03 PM, Pawel Veselov <pawel.veselov@xxxxxxxxx> wrote:

On Fri, Jul 19, 2013 at 3:54 PM, Mike Lowe <j.michael.lowe@xxxxxxxxx> wrote:
I'm not sure how to get you out of the situation you are in but what you have in your crush map is osd 2 and osd 3 but ceph starts counting from 0 so I'm guessing it's probably gotten confused.  Some history on your cluster might give somebody an idea for a fix. 

We had osd.0 and osd.1 first, then we added osd.2. We then removed osd.1, added osd.3 and removed osd.0.
Do you think that adding back a new osd.0 and osd.1, and then removing osd.2 and osd.3 will solve that confusion? I'm a bit concerned that proper osd numbering is required to maintain a healthy cluster...
 
 
On Jul 19, 2013, at 6:44 PM, Pawel Veselov <pawel.veselov@xxxxxxxxx> wrote:

Hi.

I'm trying to understand the reason behind some of my unclean pages, after moving some OSDs around. Any help would be greatly appreciated.I'm sure we are missing something, but can't quite figure out what.

[root@ip-10-16-43-12 ec2-user]# ceph health detail
HEALTH_WARN 29 pgs degraded; 68 pgs stuck unclean; recovery 4071/217370 degraded (1.873%)
pg 0.50 is stuck unclean since forever, current state active+degraded, last acting [2]
...
pg 2.4b is stuck unclean for 836.989336, current state active+remapped, last acting [3,2]
...
pg 0.6 is active+degraded, acting [3]

These are distinct examples of problems. There are total of 676 page groups.
Query shows pretty much the same on them: .

There are some pg_temps (I don't quite understand what those are), that are mapped to non-existing OSDs. osdmap: http://pastebin.com/irbRNYJz
queries for all stuck page groups:http://pastebin.com/kzYa6s2G



_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
With best of best regards
Pawel S. Veselov

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux