On 18 Jan 2007, at 05:41, Janne Peltonen wrote:
Is there documentation abt replication failover scenarios anywhere? I
can, of course, conjure up a thing or two, but I'd like to see how
other
people have resolved 'corrupted mailspool -> services to the
replica ->
maintenance -> resync master -> services back to the master'
situations.
I did a short Google, but didn't find much of notice.
Attached is our operation group's notes on the subject. It makes
reference to the tool we use to manage the OS of the machines
(radmind), but it should be pretty clear what they are talking about
without any radmind knowledge.
:wes
1. Establish primary failure
we believe that the failover procedure should take approximately 30 minutes, so the failover procedure should be invoked whenever the estimated downtime on the primary would exceed this amount of time
an exception may be made if there is reason to believe that a substantial amount of data on the failed primary was not synched to the replica; we will discuss the feasibility of sanity checks which can be run prior to failover
2. Stop cyrus/sync_client on primary if necessary / remove primary from network if necessary
/etc/init.d/cyrus stop
/etc/init.d/sync_client stop
/etc/init.d/network stop (or unplug network cable)
3. stop cyrus on the replica
/etc/init.d/cyrus stop
4. Change dns so that the name of <badhost>-repl becomes <badhost>
-> ensure you change forward and reverse
-> leave original entries commented out
5. Verify dns changes are working by checking on truelies
dig <badhost>.mail
dnsrev the ip
6. Put special files of <badhost>-repl in place for <badhost> to reflect ip information of replica
cd to special dir (generally /var/radmind/special/imap)
cp -R <badhost> <badhost>.save
cp <badhost>-repl/etc/sysconfig/network <badhost>/etc/sysconfig/network
cp <badhost>-repl/etc/sysconfig/network-devices/ifconfig.eth0 <badhost>/etc/sysconfig/network-devices
edit network to fix hostname
vi <badhost>/etc/sysconfig/network
7. radmind the replica
ra.sh update
Update command file and/or transcripts? [Yn] y
/var/radmind/client/command.K: updated
/var/radmind/client/special.T: updated
c ./dev/ttyS0 0600 0 0 4 64
special.T:
+ f ./etc/adsm/TSM.PWD 0444 0 0 1093046900 164 TIgISWWzEESwLKsM5TQx4CRH1hc=
imap/imap-23backend.T:
+ f ./etc/cyrus.conf 0644 0 0 1156541554 1380 HqMdPv649xvUptagZY1X489CCpo=
imap/imap.T:
+ f ./etc/imapd.common.conf 0644 0 0 1119845235 871 kTjkwR4x0SwRuK3qvpKi2ZGwANU=
imap/imap-23backend.T:
+ f ./etc/imapd.conf 0644 0 0 1155789187 343 RIr24APHrHa8fp6YTCezsGUCK4U=
special.T:
+ f ./etc/imapd.host.conf 0444 0 0 1156186085 104 RIgobQuTFI/HRQNmF4H4WEEoU1I=
+ f ./etc/krb5.keytab 0640 0 25 1093051728 952 hk7wwXNZgVqyiPgB8BQ55fGtULg=
+ f ./etc/sysconfig/network 0644 0 0 1166473054 81 pfuFsI4FuD763RKzCIXMHojQadc=
+ f ./etc/sysconfig/network-devices/ifconfig.eth0 0644 0 0 1166473074 78 yXkW7BokmxryTqqJKLmFl9zc3Qs=
+ f ./etc/sysconfig/network-devices/ifconfig.eth1 0444 0 0 1166473075 71 yvCcuy3ATic/4AXPPVa1zeoPnbo=
- f ./opt/tivoli/tsm/client/ba/bin/dsm.sys 0644 0 0 1164130511 418 -
+ f ./var/imap/hostname.pem 0444 0 0 1155787168 2920 Hyfrb/Sg4WkWHp/dUYHe8q9/cv4=
8. /etc/init.d/network restart
hostname <badhost> (remember to use fqdn)
pkill syslogd ksyslogd
or reboot (your choice)
9. start cyrus
su cyrus
(get tickets)
/usr/local/heimdal-k5/bin/kinit -k -l 25h imap/mail.umich.edu@xxxxxxxxx
ctl_mboxlist -m -w
(no output is good!!!)
<if ok>
(exit so you are root)
init 3
10. comment out replnag until new replica is brought up
11. restart nefu to catch ip change
*** bringing up a new replica, hopefully on same hardware **
1. update DNS for new replica
2. set up special files of <badhost>-repl
cd to special dir (generally /var/radmind/special/imap)
cp -R <badhost>-repl <badhost>-repl.save
cp <badhost>.save/etc/sysconfig/network <badhost>-repl/etc/sysconfig/network
cp <badhost>.save/etc/sysconfig/network-devices/ifconfig.eth0 <badhost>-repl/etc/sysconfig/network-devices
edit network to fix hostname
vi <badhost>-repl/etc/sysconfig/network
3. reload new replica with existing command file
4. boot new replica & start cyrus
5. generate list of mailboxes & sync
to get mailboxes
ctl_mboxlist -d > /tmp/users
awk '{ print $1 }' /tmp/users | xargs sync_client -v -l -m
6. start sync client
*** switch back during next maintenance window ***
1. stop cyrus on primary
init 2
2. verify that /var/imap/sync is empty (no pending syncs), if not run
sync_client -v -l -r -f <file>
on any remaining log files, delete each file after syncing
3. swap DNS
4. move specials back into place
cd to special dir
mv <badhost> <badhost>.old
mv <badhost>-repl <badhost>-repl.old
mv <badhost>.save <badhost>
mv <badhost>-repl.save <badhost>-repl
5. reset both machines
/etc/init.d/network restart
hostname <badhost> (remember to use fqdn)
pkill syslogd ksyslogd
or reboot (your choice)
6. start cyrus
su cyrus
(get tickets)
/usr/local/heimdal-k5/bin/kinit -k -l 25h imap/mail.umich.edu@xxxxxxxxx
ctl_mboxlist -m -w
(no output is good!!!)
<if ok>
(exit so you are root)
init 3
7. cleanup specials (remove the .old directories)
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
