Dear Patrick, > How would you do the IMAP append? Using a Perl::IMAP function? > > This isn't necessarily a concern for this list, but a few days ago I > upgraded a site from cyrus 2.1.16 to cyrus 2.3.16 by using imapsync to > transfer mail from the old server to the new one. This worked great > (i.e. all metadata seems to have been preserved), however the old server > was still collecting mail for the few hours it took to imapsync all the > users (note that this is very slows and probably not appropriate for > larger sites). My plan was to swap the servers and then do a final > imapsync from the old IMAP server to the new one. For some reason, for > some users the additional set of new messages is copied to the new > server, and others they aren't. I couldn't figure out what the > difference is, and don't want to spend too much more time on this > because of the small number of messages in play, so I need some way to > transfer a couple of dozen messages "by hand". In general, though > imapsync seems to be a great way to "clean up" cyrus folders when > switching servers. Had I known I would have sync problem later, I would > have just taken the old server off line before syncing the messages. > > Before someone suggests that I should have just copied /var/lib/cyrus > and the messages over to the new server, I didn't trust this because I > couldn't get anyone to confirm the database files I had on the old > 2.1.16 server -- the filenames were mostly not the same as this set: > > annotation_db > duplicate_db > mboxlist_db > ptscache_db > quota_db > seenstate_db > tlscache_db > > and the old server experienced a couple of devastating crashes which > required me to cyrreconstruct all the user.user mailboxes a couple of > times. It seemed pretty clear that the db files on the old server were > a mess, and upgrading from anywhere from libdb2 to libdb4.3 to libdb4.8 > seemed sketchy as well. The old server is such an old debian system > that apt-cache depends doesn't seem to work any more because the package > servers have changed. (Yeah, I could probably find the source packages > somewhere, but then there are a lot of things I could do given infinite > time.) We use IMAPsync all the time, manually and (more often) through automated programs. One of our customers uses Gmail for their corporate email system (18,000+ user accounts, about 1000-2000 accounts of ex-employees at any point in time). We recently wrote a layer on top of our mailing product for him, to download and archive all the messages of all his ex-employees' mailboxes. We used IMAPsync in an automated mode to download all his mails. He gives us the usernames and passwords, and our server-based system does the rest, creating local mirror mail accounts and downloading the Gmail mailboxes to local Cyrus IMAP mailboxes. We've learned a few things from all this. Firstly, an older version of IMAPsync (IIRC it was v1.25x) used to hang at random and consume 100% CPU indefinitely. Upgrading to a recent release (v1.34x or later) has fixed the problem. Secondly, we have found that IMAP sessions _often_ terminate halfway for no fault of IMAPsync. Reattempting the IMAP sync succeeds. Therefore, for our automated IMAP download-and-archive solution, we wrote a daemon which fires IMAPsync for each account in an infinite loop. Whenever IMAPsync terminates with errorlevel = 0, we found that we can safely believe the session completed successfully, and complete sync has been done. So our daemon makes note of this, and doesn't try to sync this user's mailboxes any more. Therefore, our IMAPsync monitor daemon is prepared to retry sync for a recalcitrant user account an infinite number of times. Once we understood that these errors are not repeatable, and hits accounts at random, we gave up trying to debug why -- we just run IMAPsync over and over till we get an all-clear. It works well. Some users had 10+GB of messages -- all of it is pulled down eventually. Of course, these random errors are probably more a Google IMAP issue than IMAP in general. If you often need to sync mailboxes among IMAP folders, I'd suggest you write a monitor daemon and fire IMAPsync from within it in an infinite loop the way I described. Shuvam ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
