Nic,
Thanks for that detailed explanation. I still feel myself somewhat
stymied by either the documentation (or lack thereof) or perhaps an
unfortunate case of being somewhat feeble-minded. Here are some follow
up comments/questions:
On 12/18/2014 9:59 AM, Nic Bernstein wrote:
> I will say that the ability to quiesce the application without halting
> it would be most desirable. Most databases have supported this sort of
> thing for ages, and it would be great if one could send a signal to
> Cyrus to achieve the same result.
I wonder what would happen if you just stopped lmtp while making a
snapshot? Would postfix choke on this and start kicking messages back
to the sender, or would they get queued for later delivery?
Alternatively, maybe lmtp could temporarily divert new messages to a
dummy spool so that postfix/sendmail wouldn't have to know anything
about this. This might be the least painful way to implement quiescence
in cyrus.
> His initial suggestion -- stop cyrus, snapshot, restart cyrus -- is
> reasonable, but we feel that the later suggestion -- stop cyrus, tar
> up data, start cyrus -- is not. It takes data offline for too long.
> That's why the snapshot capability is necessary in any truly suitable
> server.
I agree. Here is a substitute proposal (and I'll come back to why I'm
pushing this point). Serially
1. rsync user mail files
2. rsync configdirectory db files
3. rsync user mail files again
That should get you reasonably close to what you get with snapshots.
If you follow the prescribed cyrus directory structure, then this can be
simplifed (Arch linux example):
1. rsync -a --delete /var/imap/user [removable disk/other server]
2. rsync -a --delete /var/imap [removable disk/other server]
Once you've rsynced the mail files once, rsyncing them again a short
time later should be pretty fast. There does need to be a backup
solution for people who only have one server, hence can't use
replication or imapsync to do backups.
>
> Lastly, as to the use of imapsync to achieve user, mailbox or server
> replication,...
>
> So your command line is much like Patrick's example, but with '--user1
> <user> --authuser1 <proxyuser> --user2 <user>...'
> Of course you must create a proxy user, and Cyrus supports this with the
> 'proxyserver' directive in imapd.conf (man imapd.conf for details),
> i.e.: 'proxyservers: proxyuser'.
>
Here is the imapd.conf man page entry for proxyservers:
proxyservers: <none>
A list of users and groups that are allowed to proxy for other
users, separated by spaces. Any user listed in this will be
allowed to login for any other user: use with caution. In a
standard murder this option should ONLY be set on backends.
DO NOT SET on frontends or things won't work properly.
That capitalized "DO NOT SET on frontends" would seem to be cause for
concern, especially since I don't understand how this works.
For people who are
1. imapsync'ing between machines both behind a firewall
2. using saslauthd with pam
I thought of this solution: Temporarily block port 143 traffic on the
outward facing port of your firewall, and then add the line
auth sufficient pam_permit.so
to the top of /etc/pam.d/imap files on both the sending and receiving
imap servers. This should allow you to imapsync the mail stores for
every user without having to provide passwords. Once you're done,
simply remove these lines from the PAM configuration files and unblock
the port on the firewall. Yes, this will mean that users won't be able
to access their mail from outside the firewall while the imapsync is in
operation, and this is probably only workable for smaller organizations
where people are not concerned about their coworkers temporarily being
able to access their mail. There could probably be a desktop policy to
handle this as well.
However, you are 100% correct that replication would appear to be a far
less complex solution. After reading through the available
documentation, it wasn't clear to me that it was possible to do
replication without setting up a murder, a complexity I was hoping to avoid.
So, here's the feeble-mindedness component: I didn't completely follow
your explanation for setting up a replication server. It would be
awesome to have a howto for doing this -- is anyone aware of anything
like this; i.e. howto set up a replication server outside the murder
context.
> However, I must be honest and point out that if you're going to go to
> the trouble of figuring out how to use imapsync (and possibly pay for
> it, to boot) you may as well just set up a replica. As I've shown,
> above, it's just not that hard.
>
Imapsync is still useful for migrating individual users from one imap
server to another. In my case, I'm migrating from a cyrus 2.3.x server
using Berkeley db metadata files to a cyrus 2.4.x server which will be
entirely skiplist based. Understood that you can convert db files to
skiplists, but I feel most comfortable using imapsync for this. In this
use case there are only a handful of users, but they all have extremely
complex and massive mail folders.
My current plan is to use imapsync for the migration and then
replication to another dummy server for backup, assuming I can figure
out how to set up replication.
Thanks again for your helpful comments!
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
