> On Jan 16, 2019, at 12:08 PM, Anthony Verevkin <anthony@xxxxxxxxxxx> wrote: > > I would definitely see huge value in going to 3 MONs here (and btw 2 on-site MGR and 2 on-site MDS) > However 350Kbps is quite low and MONs may be latency sensitive, so I suggest you do heavy QoS if you want to use that link for ANYTHING else. > If you do so, make sure your clients are only listing the on-site MONs so they don't try to read from the off-site MON. > Still you risk the stability of the cluster if the off-site MON starts lagging. If it's still considered on while lagging, all changes to cluster (osd going up/down, etc) would be blocked by waiting it to commit. Using QOS is definitely something I hadn’t thought of, thanks. Setting up tc wouldn’t be rocket science. I’d probably make sure that the offsite mon wasn’t even reachable from clients, just the other mons. > Even if you choose against an off-site MON, maybe consider 2 on-site MON instead. Yes, you'd double the risk of cluster going to a halt if any one node dies vs one specific node dying. But if that happens you have a manual way of downgrading to a single MON (and you still have your MON's data) vs risking to get stuck with a OSD-only node that had never had MON installed and not having a copy of MON DB. I had thought through some of that. That’s a good idea of having it ready to go though. It got also me thinking if it would be practical to have the two local and one remote set up and ready to go as you note, but only two running at a time. So if I need to take a primary node down, I would re-add the remote, do the service on the primary node, bring it back, re-establish mon health, then remove the remote. That’s probably great until there’s an actual primary failure — no quorum and the out-of-date remote can’t be re-added so one is just as bad off. > I also see how you want to get the data out for backups. > Having a third replica off-site definitely won't fly with such bandwidth as it would once again block the IO until committed by the off-site OSD. > I am not quite sure RBD mirroring would play nicely with this kind of link either. Maybe stick with application-level off-site backups. > And again, whatever replication/backup strategy you do, need to QoS or else you'd cripple your connection which I assume is used for some other communications as well. The connection is a consumer-grade gig fiber. It would be great if I could optimize it for higher speeds, the locations are only 30 miles from each other! It seems at this point that I’m not missing anything, I’m grateful for your thoughts! I think I just need to get another node in there, whether through a VPS from the provider or another unit. The cost of dealing with the house of cards being built seems much higher as soon as there is a single unplanned configuration and hours get put into bringing it back to health. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |