On Wed, Sep 30, 2009 at 01:01:28AM -0400, Brian Awood wrote: > On Tuesday 29 September 2009 @ 18:41, Bron Gondwana wrote: > > > > Possibly the secret is that we use IPAddr2 from linux-ha to force > > ARP flushes, and we transfer the primary IP address between > > machines, so nothing else needs to know - we just shut down one end > > and bring up the other with the IP and it's all good. > > Our primaries and replicas are located in different data centers, and > since we have not control over how the network is subdivided it's > impossible for them to take the same IPs. Yeah, fair enough. > > Our process is: > > > > a) check there are less than 10kb of files in $conf/sync/ - else > > abort b) shut down the master (host A) > > c) run sync_client -f $file on each file in $conf/sync (if any) > > c2) (if any sync fails, restart the master (host A)) > > d) shut down the replica (host B) > > e) update the database with the new master location > > f) start up the replica (host A) > > g) start up the master (host B) > > > > This means replication starts immediately, because the replica is > > already there when the master starts. > > So you just immediately start replicating back to a host (or site) > that just failed? How does that work? We don't usually "fail" as such. We're transferring the master role to a different machine. Generally you have some advance warning something bad is happening (like, a single drive in a RAIDset fails) and transition the master to the less-risky location until the RAID has rebuilt. Or you're doing maintainence on the machine that had the master role. Sure - in a host death situation there's a "force" mode which just does the "host B" parts. Then you have to figure out what needs fixing semi-manually afterwards. We don't have a cleaner-upperer yet. We do have "checkreplication" though, which does a pretty good job of finding what's wrong between the two machines. > We have a third level of machines that we sync to, in an out of band > process, but the data is stored exactly the same way so we can start > replicating to them immediately. So even if a entire data center > failed, we can still be running a fully replicated service with > almost no downtime visible to users. Yeah, that would be nice. We don't have a second datacentre at the moment. We're planning to get one running at some point, at least for the higher-level of paying customers! Bron. ---- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
