The mons work best when they know absolutely everything. If they know that osd.3 was down 40 seconds before osd.2 that means that if a write was stilling happening while osd.2 was still up, the mons have record of it in the maps and when osd.3 comes up, it can get what it needs from the other osds. Mons are the keepers of maps, epochs, and everything important to know about a cluster. If you're using encryption on your OSDs, the mons keep track of the keys to decrypt the osds iirc.
Even if you aren't using encryption, the OSDs check with the Mons when they first start to know what the most recent map is. If they can't communicate with a Mon, they will fail to start and die. Last down/first up ensures that the mons know everything and is the safest way to handle a cluster shutdown. Yes Ceph can usually handle full system power-offs with no proper order or having too many of something shutdown while the rest of the cluster is running, but most people try to avoid disaster scenarios if they can help it.
On Mon, May 7, 2018 at 9:48 PM Bryan Henderson <bryanh@xxxxxxxxxxxxxxxx> wrote:
There is a lot of advice around on shutting down a Ceph cluster that says
to shut down the OSDs before the monitors and bring up the monitors before
the OSDs, but no one explains why.
I would have thought it would be better to shut down the monitors first and
bring them up last, so they don't have to witness all the interim states with
OSDs down. And it should make the noout, nodown, etc. settings unnecessary.
So what am I missing?
Also, how much difference does it really make? Ceph is obviously designed to
tolerate any sequence of failures and recoveries of nodes, so how much risk
would I be taking if I just haphazardly killed everything instead of
orchestrating a shutdown?
--
Bryan Henderson San Jose, California
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
