How to replace an node in ceph?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



2014-09-05 13:19 GMT+08:00 Christian Balzer <chibi at gol.com>:

>
> Hello,
>
> On Fri, 5 Sep 2014 12:09:11 +0800 Ding Dinghua wrote:
>
> > Please see my comment below:
> >
> >
> > 2014-09-04 21:33 GMT+08:00 Christian Balzer <chibi at gol.com>:
> >
> > >
> > > Hello,
> > >
> > > On Thu, 4 Sep 2014 20:56:31 +0800 Ding Dinghua wrote:
> > >
> > > Aside from what Loic wrote, why not replace the network controller or
> > > if it is onboard, add a card?
> > >
> > > > Hi all,
> > > >         I'm new to ceph, and apologize if the question has been
> > > > asked.
> > > >
> > > >         I have setup a 8-nodes ceph cluster, and after two months
> > > > running, network controller of an node is broken, so I have to
> > > > replace the node with an new one.
> > > >         I don't want to trigger data migration, since all I want to
> > > > do is replacing a node, not shrink the cluster and then enlarge the
> > > > cluster.
> > >
> > > Well, you will have (had) data migration unless your cluster was set to
> > > noout from the start or had a "mon osd downout subtree limit" set
> > > accordingly.
> > >
> >   [Ding Dinghua]: Yes, I have already set noout flag
> >
> > >
> > > >         I think the following steps may work:
> > > >         1)  set osd_crush_update_on_start to false, so when osd
> > > > starts, it won't modify crushmap and trigger data migration.
> > > I think the noin flag might do that trick, too.
> > >
> >  [Ding Dinghua]: I set osd_crush_update_on_start to false, so when the
> > osds on the new node start,
> >                         /etc/init.d/ceph script won't do "ceph osd crush
> > create-or-move", and the osds on new node will still in the old host, so
> > no data migration will occur.
> >
> > > >           2)  set noout flags to prevent osds been kicked out of
> > > > cluster and trigger data migration
> > > Probably too late at this point...
> > >
> > > >           3)  mark all osds on the broken node down(actually, since
> > > > network controller is broken, these osds are already down)
> > > And not out?
> > >
> >   [Ding Dinghua]: Yes, since noout flag is set, these osds are [down, in]
> >
> So far, so good.
>
> However see below:
>
> > >
> > > Regards,
> > >
> > > Christian
> > > >           4)  prepare osd on the new node, and keep osd_num the same
> > > > with the osd on the broken node:
> > > >                ceph-osd -i [osd_num] --osd-data=path1 --mkfs
>
> I don't think that will work. To recycle OSDs they would have to be removed
> (triggering migration) first.
>
 [Ding Dinghua]:  I'm not removing/creating osds,  just change the storage
which holds the osds, so osd id on the broken node should be used.

> Just adding new OSDs should do the trick, though.
>
[Ding Dinghua]: It will trigger data migration

>
> Christian
>
> > > >         5) start osd on the new node, and peering and backfilling
> > > > work will be started automaticlly
> > > >         6)  wait until 5) complete, and repeat 4) and 5) until all
> > > > osds on the broken node been moved to the new node
> > > >         I have done some test on my test cluster, and it seemed
> > > > works, but I'm not quite sure it's right in theory, so any comments
> > > > will be appreciated.
> > > >         Thanks.
> > > >
> > >
> > >
> > > --
> > > Christian Balzer        Network/Systems Engineer
> > > chibi at gol.com           Global OnLine Japan/Fusion Communications
> > > http://www.gol.com/
> > >
> >
> >
> >
>
>
> --
> Christian Balzer        Network/Systems Engineer
> chibi at gol.com           Global OnLine Japan/Fusion Communications
> http://www.gol.com/
>



-- 
Ding Dinghua
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140905/dc8b57d7/attachment.htm>


[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux