In addition, from my experience: I often set noout, norebalance and nobackfill before doing maintenance. This greatly speeds up peering (when adding new OSDs) and reduces unnecessary load from all daemons. In particular, if there is heavy client IO going on at the same time, the ceph daemons are much more stable with these settings. I had, after shutting down one host, more OSDs crashing under combined peering+backfill load causing a cascade of even more OSDs crashing. The above settings have prevented such things from happening. As mentioned before, it also avoids unnecessary rebuilds of objects that are not even modified during the service window. Having an OSD down even for 30 minutes usually requires only a few seconds to minutes to catch up with the latest diffs of modified objects instead of starting a full rebuild of all objects regardless of their modification state. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Etienne Menguy <etienne.menguy@xxxxxxxx> Sent: 22 September 2021 12:17:39 To: ceph-users Subject: Re: Why set osd flag to noout during upgrade ? Hello, >From my experience, I see three reasons : - You don’t want to recover data if you already have them on a down OSD, rebalancing can have a big impact on performance - If upgrade/maintenance goes wrong you will want to focus on this issue and not have to deal with things done by Ceph meanwhile. - During an upgrade you have an ‘unusual’ cluster with different versions. It’s supposed to work, but you probably want to keep it ‘boring’. - Etienne Menguy etienne.menguy@xxxxxxxx > On 22 Sep 2021, at 11:51, Francois Legrand <fleg@xxxxxxxxxxxxxx> wrote: > > Hello everybody, > > I have a "stupid" question. Why is it recommended in the docs to set the osd flag to noout during an upgrade/maintainance (and especially during an osd upgrade/maintainance) ? > > In my understanding, if an osd goes down, after a while (600s by default) it's marked out and the cluster will start to rebuild it's content elsewhere in the cluster to maintain the redondancy of the datas. This generate some transfer and load on other osds, but that's not a big deal ! > > As soon as the osd is back, it's marked in again and ceph is able to determine which data is back and stop the recovery to reuse the unchanged datas which are back. Generally, the recovery is as fast as with noout flag (because with noout, the data modified during the down period still have be copied to the back osd). > > Thus is there an other reason apart from limiting the transfer and others osds load durind the downtime ? > > F > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |