There have been threads on exactly this. Might depend a bit on your ceph version. We are running mimic and have no issues doing: - set noout, norebalance, nobackfill - add all OSDs (with weight 1) - wait for peering to complete - unset all flags and let the rebalance loose Starting with nautilus there seem to be issues with this procedure. Mainly the peering phase can cause a collapse of the cluster. In your case, it sounds like you added the OSDs already. You should be able to do relatively safely: - set noout, norebalance, nobackfill - set weight of OSDs to 1 one by one and wait for peering to complete every time - unset all flags and let the rebalance loose I believe once the peering succeeded without crashes, the rebalancing will just work fine. You can easily control how much rebalancing is going on. I noted that ceph seems to have a strange concept of priority though. I needed to gain capacity by adding OSDs and ceph was very consequent with moving PGs from the fullest OSDs last. The opposite of what should happen. Thus, it took ages for additional capacity to become available and also the backfill too full warnings stayed for all the time. You can influence this to some degree by using force_recovery commands on PGs on the fullest OSDs. Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kristof Coucke <kristof.coucke@xxxxxxxxx> Sent: 21 October 2020 13:29:00 To: ceph-users@xxxxxxx Subject: Question about expansion existing Ceph cluster - adding OSDs Hi, I have a cluster with 182 OSDs, this has been expanded towards 282 OSDs. Some disks were near full. The new disks have been added with initial weight = 0. The original plan was to increase this slowly towards their full weight using the gentle reweight script. However, this is going way too slow and I'm also having issues now with "backfill_toofull". Can I just add all the OSDs with their full weight, or will I get a lot of issues when I'm doing that? I know that a lot of PGs will have to be replaced, but increasing the weight slowly will take a year at the current speed. I'm already playing with the max backfill to increase the speed, but every time I increase the weight it will take a lot of time again... I can face the fact that there will be a performance decrease. Looking forward to your comments! Regards, Kristof _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |