Hi Matt,
Also, make sure that when rejoining host has correct time. I have seen
clusters going down when rejoining hosts that were down for maintenance
for various weeks and came in with datetime deltas of some months (no
idea why that happened, I arrived with the firefighter team ;-) )
Cheers
El 27/11/22 a las 13:27, Frank Schilder escribió:
Hi Matt,
if you didn't touch the OSDs on that host, they will join and only objects that have been modified will actually be updated. Ceph keeps some basic history information and can detect changes. 2 weeks is not a very long time. If you have a lot of cold data, re-integration will go fast.
Initially, you will see a huge amount of misplaced objects. However, this count will go down much faster than objects/s recovery.
Before you rejoin the host, I would fix its issues though. Now that you have it out of the cluster, do the maintenance first. There is no rush. In fact, you can buy a new host, install the OSDs in the new one and join that to the cluster with the host-name of the old host.
If you consider replacing the host and all disks, the get a new host first and give it the host name in the crush map. Just before you deploy the new host, simply purge all down OSDs in its bucket (set norebalance) and deploy. Then, the data movement is restricted to re-balancing to the new host.
If you just want to throw out the old host, destroy the OSDs but keep the IDs intact (ceph osd destroy). Then, no further re-balancing will happen and you can re-use the OSD ids later when adding a new host. That's a stable situation from an operations point of view.
Hope that helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Matt Larson<larsonmattr@xxxxxxxxx>
Sent: 26 November 2022 21:07:41
To: ceph-users
Subject: What to expect on rejoining a host to cluster?
Hi all,
I have had a host with 16 OSDs, each 14TB in capacity that started having
hardware issues causing it to crash. I took this host down 2 weeks ago,
and the data rebalanced to the remaining 11 server hosts in the Ceph
cluster over this time period.
My initial goal was to then remove the host completely from the cluster
with `ceph osd rm XX` and `ceph osd purge XX` (Adding/Removing OSDs — Ceph
Documentation
<https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/>).
However, I found that after the large amount of data migration from the
recovery, that the purge and removal from the crush map for an OSDs still
required another large data move. It appears that it would have been a
better strategy to assign a 0 weight to an OSD to have only a single larger
data move instead of twice.
I'd like to join the downed server back into the Ceph cluster. It still
has 14 OSDs that are listed as out/down that would be brought back online.
My question is what can I expect if I bring this host online? Will the
OSDs of a host that has been offline for an extended period of time and out
of the cluster have PGs that are now quite different or inconsistent? Will
this be problematic?
Thanks for any advice,
Matt
--
Matt Larson, PhD
Madison, WI 53705 U.S.A.
_______________________________________________
ceph-users mailing list --ceph-users@xxxxxxx
To unsubscribe send an email toceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list --ceph-users@xxxxxxx
To unsubscribe send an email toceph-users-leave@xxxxxxx
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
