As far as i know you can do it in two ways: (assuming you hace a pool size of 3 on all 3 nodes with min_size 2 to still have access to data)
1. Set noout for not starting the rebalance of the cluster. Reinstall OS on the faulty node and redeploy the node with all keys and conf files (either manual or with ceph-deploy) without rezapping the OSD's. Activate the OSD's and unset the noout. This should be enogh, because the OSD have the same cluster UUID.
2. Set pool size to 2 and remove the OSD's (before setting an OSD out, first reweigth it to 0, then set it out than remove from cluster)from the faulty node from crush map. (wait for rebalance if any). Reinstall OS on the node and redeploy ceph on that node creating new OSDs in that node (they should get the same id as before).
I have successfully done the first one a while back and it worked, although i don't remember the exat commands i did.
2016-02-10 19:29 GMT+02:00 Ivan Grcic <igrcic@xxxxxxxxx>:
Hi Daniel,
oops, wrong copy paste, here are the correct commands:
ceph osd pool get pool-name size
ceph osd pool set pool-name size 2
On Wed, Feb 10, 2016 at 6:27 PM, Ivan Grcic <igrcic@xxxxxxxxx> wrote:
> Grüezi Daniel,
>
> my first question would be: Whats your pool size / min_size?
>
> ceph osd pool get pool-name
>
> It is probably 3 (default size). If you want to have healthy state
> again with only 2 nodes (all the OSDs on node 3 are down), you have to
> set your pool size to 2:
>
> ceph osd pool set pool-name 2
>
> Regards,
> Ivan
>
> On Wed, Feb 10, 2016 at 5:46 PM, <Daniel.Balsiger@xxxxxxxxxxxx> wrote:
>> Hi Ceph users
>>
>> This is my first post on this mailing list. Hope it's the correct one. Please redirect me to the right place in case it is not.
>> I am running a small (3 nodes with 3 OSD and 1 monitor on each of them) Ceph cluster.
>> Guess what, it is used as Cinder/Glance/Nova RDB storage for OpenStack.
>>
>> I already replaced some single OSD (faulty disk) without any problems.
>> Now I am facing another problem since the system disk on one of the 3 nodes failed.
>> So I thought to take the 3 OSDs of this node out of the cluster, set up the node from scratch and add the 3 OSDs again.
>>
>> I did successfully take out the first 2 OSDs.
>> Yes I hit the corner case, I did it with " ceph osd crush reweight osd.<OSD#>0.0", waited for active+clean and followed by "ceph osd out <OSD#>"
>> Status is now:
>>
>> cluster d1af2097-8535-42f2-ba8c-0667f90cab61
>> health HEALTH_WARN
>> too many PGs per OSD (329 > max 300)
>> 1 mons down, quorum 0,2 ceph0,ceph2
>> monmap e1: 3 mons at {ceph0=10.0.0.30:6789/0,ceph1=10.0.0.31:6789/0,ceph2=10.0.0.32:6789/0}
>> election epoch 482, quorum 0,2 ceph0,ceph2
>> osdmap e1628: 9 osds: 9 up, 7 in
>> pgmap v2187375: 768 pgs, 3 pools, 38075 MB data, 9129 objects
>> 119 GB used, 6387 GB / 6506 GB avail
>> 768 active+clean
>>
>> HEALTH_WARN is because of 1 monitor down (broken node) and too many PGs per OSD (329 > max 300), since I am removing OSDs
>>
>> Now the problem I am facing: When I try to reweight the 3rd OSD to 0 the cluster never comes to the active+clean state anymore.
>> # ceph --version
>> ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
>> # ceph osd crush reweight osd.7 0.0
>> reweighted item id 7 name 'osd.7' to 0 in crush map
>> # ceph -s
>> cluster d1af2097-8535-42f2-ba8c-0667f90cab61
>> health HEALTH_WARN
>> 768 pgs stuck unclean
>> recovery 817/27387 objects degraded (2.983%)
>> recovery 9129/27387 objects misplaced (33.333%)
>> 1 mons down, quorum 0,2 ceph0,ceph2
>> monmap e1: 3 mons at {ceph0=10.0.0.30:6789/0,ceph1=10.0.0.31:6789/0,ceph2=10.0.0.32:6789/0}
>> election epoch 482, quorum 0,2 ceph0,ceph2
>> osdmap e1682: 9 osds: 9 up, 7 in; 768 remapped pgs
>> pgmap v2187702: 768 pgs, 3 pools, 38076 MB data, 9129 objects
>> 119 GB used, 6387 GB / 6506 GB avail
>> 817/27387 objects degraded (2.983%)
>> 9129/27387 objects misplaced (33.333%)
>> 768 active+remapped
>>
>> I also remarked I need to reweight to 0.7 to get in active+clean state again.
>>
>> Any idea how to remove this last OSD that I can setup the node again ?
>> Thank you in advance, any help appreciated.
>>
>> Daniel
>>
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
