Thank you Frank and Eneko, Without help and support from ceph admins like you, I would be adrift. I really appreciate this. I rejoined the host now one week ago, and the cluster has been dealing with the misplaced objects and recovering well. I will use this strategy in the future: "If you consider replacing the host and all disks, get a new host first and give it the host name in the crush map. Just before you deploy the new host, simply purge all down OSDs in its bucket (set norebalance) and deploy. Then, the data movement is restricted to re-balancing to the new host. If you just want to throw out the old host, destroy the OSDs but keep the IDs intact (ceph osd destroy). Then, no further re-balancing will happen and you can re-use the OSD ids later when adding a new host. That's a stable situation from an operations point of view." Last question I have is that I am now seeing that some OSDs have uneven load of PGs, which balancer do you recommend and any caveats for how the balancer operations can affect/slow the cluster? Thanks, Matt On Mon, Nov 28, 2022 at 2:23 AM Eneko Lacunza <elacunza@xxxxxxxxx> wrote: > Hi Matt, > > Also, make sure that when rejoining host has correct time. I have seen > clusters going down when rejoining hosts that were down for maintenance for > various weeks and came in with datetime deltas of some months (no idea why > that happened, I arrived with the firefighter team ;-) ) > > Cheers > > El 27/11/22 a las 13:27, Frank Schilder escribió: > > Hi Matt, > > if you didn't touch the OSDs on that host, they will join and only objects that have been modified will actually be updated. Ceph keeps some basic history information and can detect changes. 2 weeks is not a very long time. If you have a lot of cold data, re-integration will go fast. > > Initially, you will see a huge amount of misplaced objects. However, this count will go down much faster than objects/s recovery. > > Before you rejoin the host, I would fix its issues though. Now that you have it out of the cluster, do the maintenance first. There is no rush. In fact, you can buy a new host, install the OSDs in the new one and join that to the cluster with the host-name of the old host. > > If you consider replacing the host and all disks, the get a new host first and give it the host name in the crush map. Just before you deploy the new host, simply purge all down OSDs in its bucket (set norebalance) and deploy. Then, the data movement is restricted to re-balancing to the new host. > > If you just want to throw out the old host, destroy the OSDs but keep the IDs intact (ceph osd destroy). Then, no further re-balancing will happen and you can re-use the OSD ids later when adding a new host. That's a stable situation from an operations point of view. > > Hope that helps. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Matt Larson <larsonmattr@xxxxxxxxx> <larsonmattr@xxxxxxxxx> > Sent: 26 November 2022 21:07:41 > To: ceph-users > Subject: What to expect on rejoining a host to cluster? > > Hi all, > > I have had a host with 16 OSDs, each 14TB in capacity that started having > hardware issues causing it to crash. I took this host down 2 weeks ago, > and the data rebalanced to the remaining 11 server hosts in the Ceph > cluster over this time period. > > My initial goal was to then remove the host completely from the cluster > with `ceph osd rm XX` and `ceph osd purge XX` (Adding/Removing OSDs — Ceph > Documentation<https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/> <https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/>). > However, I found that after the large amount of data migration from the > recovery, that the purge and removal from the crush map for an OSDs still > required another large data move. It appears that it would have been a > better strategy to assign a 0 weight to an OSD to have only a single larger > data move instead of twice. > > I'd like to join the downed server back into the Ceph cluster. It still > has 14 OSDs that are listed as out/down that would be brought back online. > My question is what can I expect if I bring this host online? Will the > OSDs of a host that has been offline for an extended period of time and out > of the cluster have PGs that are now quite different or inconsistent? Will > this be problematic? > > Thanks for any advice, > Matt > > -- > Matt Larson, PhD > Madison, WI 53705 U.S.A. > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > Eneko Lacunza > Zuzendari teknikoa | Director técnico > Binovo IT Human Project > > Tel. +34 943 569 206 | https://www.binovo.es > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun > https://www.youtube.com/user/CANALBINOVOhttps://www.linkedin.com/company/37269706/ > > -- Matt Larson, PhD Madison, WI 53705 U.S.A. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |