|
Folks, Following Willy Offermans' recent response to this thread, I've decided to jump in, but wish to quote a bit more of Patrick's original message (below), hence the out-of-sequence reply. I'd like to address Patrick's issues in pretty much the order they're presented here. Firstly, however, I would like to echo some confusion as to the purpose of cyrdump. It seems to have been around since at least 2.2.3, but documentation has appeared only fleetingly, and says little about the program:
Not much help, that.As for the utility of filesystem snapshots, be they LVM2, ZFS, XFS, etc., this of course extends only so far as the stability of the underlying data. If your data is in flux at the time of the snapshot (unwritten buffers, etc.) then you'll only snapshot an incoherent, or "crash-consistent", state. Thus it's important to consider the relative cost and consequence of taking the extra steps to quiesce the application prior to the snapshot. We perform ZFS snapshots very frequently, without application quiescence, under the belief that even crash-consistent data is better than no data, in a DR scenario, and we replicate this data across multiple machines and sites, to further improve any potential DR outcome. However, we also make "application-consistent" backups, which require application quiescence, as Patrick has described. His initial suggestion -- stop cyrus, snapshot, restart cyrus -- is reasonable, but we feel that the later suggestion -- stop cyrus, tar up data, start cyrus -- is not. It takes data offline for too long. That's why the snapshot capability is necessary in any truly suitable server. If you, like us, maintain your "config" and spool (and potentially meta) data on separate filesystems, you must remember to snapshot them all. Also, if you're going to stop and start cyrus like this, it pays to remove heavyweight processes (i.e. delprune, checkpoint, sqautter) from the START section of cyrus.conf, to prevent extensive delays at restart, and let them be performed periodically (in EVENTS) instead. I will say that the ability to quiesce the application without halting it would be most desirable. Most databases have supported this sort of thing for ages, and it would be great if one could send a signal to Cyrus to achieve the same result. One wonders, however, if the model of master process (cyrmaster for Debian-based distros) with potentially hundreds or thousands of child daemons, would lend itself to this. But snapshots, whatever their consistency or coherence, will always be less useful for DR than a replica, and in Cyrus this is not only available, but not really that hard to get up and running. The differences between the imapd.conf files on a master and replica can be this simple, or simpler even, as this example is from a murder (from a running pair): The changes in cyrus.conf are similarly brief:--- imapd-master.conf 2014-11-24 13:37:45.752639666 -0600 +++ imapd-replica.conf 2014-11-24 13:31:58.702611566 -0600 @@ -1,22 +1,14 @@ ## -# These configuration parameters are for the master server -# in a replication set -servername: master.example.com +# These configuration parameters are for the replica server in a +# replication cluster +servername: replica.example.com -## -# Auth credentials -mupdate_server: mupdate.example.com -mupdate_username: mupdate -mupdate_authname: mupdate -mupdate_password: <password> - -## -# Replication support -# This is how the BACKEND for this host is defined -sync_host: replica.example.com -sync_authname: mailproxy -sync_password: <password> -sync_compress: true -sync_log: true -sync_repeat_interval: 5 -sync_shutdown_file: /var/run/cyrus/sync_stop +## Auth credentials +# The credentials below must match the account listed in lmtp_admins +# on the backend servers. +proxy_authname: mailproxy +proxy_password: <password> +force_sasl_client_mech: PLAIN +mupdate_mechs: PLAIN +mailbox_mechs: PLAIN +serverlist: master It's just not that hard to implement, and for the security and safety it provides is well worth it.--- master/cyrus.conf 2014-11-24 14:01:26.832602819 -0600 +++ replica/cyrus.conf 2014-11-24 14:01:42.212637226 -0600 @@ -15,8 +15,8 @@ # Master sends mailbox updates to mupdate. # Replication client runs on Master. # Comment these 2 lines out on replicas - #mupdatepush cmd="/usr/sbin/cyrus ctl_mboxlist -m" - #syncclient cmd="/usr/lib/cyrus/bin/sync_client -r" + mupdatepush cmd="/usr/sbin/cyrus ctl_mboxlist -m" + syncclient cmd="/usr/lib/cyrus/bin/sync_client -r" # this is recommended if using duplicate delivery suppression #delprune cmd="/usr/sbin/cyrus expire -E 3 -D 60 -X 60" at=0400 @@ -55,7 +55,7 @@ ## # Synchronization for remote replication. # Comment this out on Master, uncomment on replicas - syncserver cmd="/usr/lib/cyrus/bin/sync_server" listen="csync" + #syncserver cmd="/usr/lib/cyrus/bin/sync_server" listen="csync" # Experimental httpd for caldav httpd cmd="httpd" listen=8080 prefork=1 maxchild=20 Note that if you're using a murder, the server name must be preserved between the two hosts, in the event of a fail-over, and only the master should communicate with the mupdate server (as shown in my example). If you're not using a murder, then most of these differences go away (all the mupdate stuff). Of course, you'll need to switch DNS or have some other way of dealing with fail-over on layer 3. We put all common configuration into /etc/imapd.conf, and then use the @include directive to include the appropriate file fragment, either imapd-master.conf or imapd-replica.conf. For the cyrus.conf, it's just a couple of line edits -- commenting and uncommenting -- so we do that by hand. You've got to intervene to update the DNS, in any event, so this much more is trivial. Lastly, as to the use of imapsync to achieve user, mailbox or server replication, and the authentication requirements thereof, I would suggest reading the README file for that application, which includes this: You may authenticate as one user (typically an admin user), but be
authorized as someone else, which means you don't need to know every
user's personal password. Specify --authuser1 "adminuser" to enable this
on host1.
So your command line is much like Patrick's example, but with
'--user1 <user> --authuser1 <proxyuser> --user2
<user>...'Of course you must create a proxy user, and Cyrus supports this with the 'proxyserver' directive in imapd.conf (man imapd.conf for details), i.e.: 'proxyservers: proxyuser'. However, I must be honest and point out that if you're going to go to the trouble of figuring out how to use imapsync (and possibly pay for it, to boot) you may as well just set up a replica. As I've shown, above, it's just not that hard. Cheers, -nic On 12/15/2014 05:01 PM, Patrick Goetz
wrote:
-- Nic Bernstein nic@xxxxxxxxxxx Onlight, Inc. www.onlight.com 219 N. Milwaukee St., Suite 2a v. 414.272.4477 Milwaukee, Wisconsin 53202 |
begin:vcard fn:Nic Bernstein n:Bernstein;Nic org:Onlight, Inc. adr:Suite 2A;;219 N Milwaukee St.;Milwaukee;WI;53202;USA email;internet:nic@xxxxxxxxxxx title:VP Operations tel;work:414-272-4477 x204 tel;cell:414-807-1734 url:http://www.onlight.com/ version:2.1 end:vcard
---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
