Hello, I would have risked a nodown option for this short downtime. We had a similar experience when we updated a bonded switch and had reboot it. Some of the connections dropped and whole cluster started marking some osds as down. Due to this almost all osd were marked as down, but none of the processes stopped. When we rebooted next switch, we used a nodown and there were no flapping osds at all that time. Best of luck next time. On Fri, Oct 5, 2018 at 4:03 PM Martin Palma <martin@xxxxxxxx> wrote: > > Thank you all for the clarification and suggestion. > > Here is a small experience report what happened during the network > maintenance, maybe it is useful for others too: > > As previously written the Ceph cluster is stretched across two data > centers and has a size of 39 storage nodes with a total of 525 OSDs > and 5 monitor nodes. > > The problem: Due to a network maintenance the connection between the > two data center will be down for approximately 8-15 seconds, which > will affect the Ceph cluster's public network. > > Before the maintenance we set the following flags: > "noout, nobackfill, norebalance, noscrub, nodeep-scrub" > > During maintenance: The network between the two data centers was down > for a total of 12 seconds. At first, it seemed everything worked fine. > Some OSDs where marked as down but came back quickly and monitor nodes > started a new election. But then more and more OSDs where marked as > down wrongly, in reality, their process was up and they had network > connectivity. Moreover, two monitors of one data center couldn't join > the quorum anymore. The network team quickly figured out what the > problem was: A wrong MTU size. After they fixed it the two monitor > nodes rejoined the quorum and nearly all OSDs came up again. Only 36 > OSD remained down and by checking them it revealed they were really > down. After a total time of 40 minutes, the cluster reached a healthy > state again. No data loss. > > Best, > Martin > On Thu, Oct 4, 2018 at 11:09 AM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote: > > > > Mons are also on a 30s timeout. > > Even a short loss of quorum isn‘t noticeable for ongoing IO. > > > > Paul > > > > > Am 04.10.2018 um 11:03 schrieb Martin Palma <martin@xxxxxxxx>: > > > > > > Also monitor election? That is the most fear we have since the monitor > > > nodes will no see each other for that timespan... > > >> On Thu, Oct 4, 2018 at 10:21 AM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote: > > >> > > >> 10 seconds is far below any relevant timeout values (generally 20-30 seconds); so you will be fine without any special configuration. > > >> > > >> Paul > > >> > > >> Am 04.10.2018 um 09:38 schrieb Konstantin Shalygin <k0ste@xxxxxxxx>: > > >> > > >>>> What can we do of best handling this scenario to have minimal or no > > >>>> impact on Ceph? > > >>>> > > >>>> We plan to set "noout", "nobackfill", "norebalance", "noscrub", > > >>>> "nodeep", "scrub" are there any other suggestions? > > >>> > > >>> ceph osd set noout > > >>> > > >>> ceph osd pause > > >>> > > >>> > > >>> > > >>> k > > >>> > > >>> _______________________________________________ > > >>> ceph-users mailing list > > >>> ceph-users@xxxxxxxxxxxxxx > > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |