I have been running Ceph over a gigabit WAN for a few months now and have been happy with it. Mine is set up with Strongswan tunnels And dynamic routing with BIRD) (although I would have used transport Mode and iBGP in hindsight). I generally have 300-500kbps flow with 5ms latency. What I specifically make sure of is the osds and clients are all on one side of the connection. That may seem odd, but it allows me to run a HA cluster with only two machines on one end. And if you think through the failure configurations is that line goes down, they pretty much all work if two machines are up and you don’t do maintenance on them during thunderstorms :-). Sent from my iPhone > On May 19, 2020, at 11:34, Benjeman Meekhof <bmeekhof@xxxxxxxxx> wrote: > > It is possible to run a ceph cluster over a WAN if you have reliable > enough WAN with sites close enough for low-ish latency. The OSiRIS > project is architected that way with Ceph services spread evenly > across three university sites in Michigan. There's more information > and contact on their website: http://www.osris.org > > We had a variety of interesting WAN outages which Ceph has always > handled well in terms of not losing data or our cluster definitions. > Outages were at times further complicated by inconsistent pathing for > cluster and backend networks such that only one or the other might be > up to some sites. In all that, with 3 mons situated 1 per site, we > never encountered any kind of split brain situations. > > Though I'm no longer involved the project is still ongoing and I'm > sure if you want to reach out they (or I personally) would be happy to > answer any questions. > > thanks, > Ben > >> On Tue, May 19, 2020 at 1:03 PM Nathan Fish <lordcirth@xxxxxxxxx> wrote: >> >> It is my understanding that it refers to running a single, normal ceph >> cluster with it's component hosts connected over WAN. This would >> require OSDs to connect to other OSDs and mons over WAN for nearly >> every operation, and is not likely to perform acceptably. >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |