Hi folks,
We have many users who run Deis on AWS, and our default configuration places hosts in an autoscaling group. Ceph runs on all hosts in the cluster (monitors and OSDs), and users have reported losing quorum after having several autoscaling events (new nodes getting added, old nodes terminated). This makes sense, as the monitor map is piled up with old entries, and eventually Ceph thinks there aren't enough healthy monitors to maintain quorum.
I know that permanently losing hosts fairly frequently is likely uncommon for most Ceph users, but I was hoping someone might have some ideas as to how best to combat this. I've been thinking about this in https://github.com/deis/deis/issues/2877, and I've come to the conclusion that the clearest path forward right now is to build a service which interacts with the Ceph API and keeps an eye on quorum state. When losing another host would prevent Ceph from achieving quorum, the service would remove a monitor from the monitor map and mark its OSD as out, ensure the placement groups are replicated elsewhere (we have min_size = 3 so the dead OSD has nothing we need), and then remove the OSD.
Is this the best solution? Are there any configuration options I've missed that we could use in the interim, such as automatically considering a monitor/OSD as dead if it's been offline for a certain amount of time?
Any tips are helpful.
Thanks!
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
