Quoting Andrew Morgan <morgan@xxxxxxxx>:
On Mon, 15 Dec 2014, Patrick Goetz wrote:On 12/10/2014 03:47 AM, Willy Offermans wrote:I'm not sure what you mean with ``all the metadata'', but there are user flags saved into the cyrdump file as well. I never performed the whole cycle of dump and restore (probably nobody did so far), so I cannot tell you that all metadata is available in the dump file. See my question above!A while back I was working on an email server that (unbeknownst to me) was connected to a UPS, but with an external disk array that was plugged in to an outlet on the UPS that was not battery-backed. This site had frequent power problems, so it turns out that power cycling a disk array while the server stays up is an awesome way to corrupt your entire file system. Since I didn't know what I was doing at the time, I restored partition-default: /home/cyrus without also restoring configdirectory: /var/lib/cyrus I was consequently confused when no mailboxes showed up, and had to then learn about and use reconstruct -r on each individual mailbox (cyrreconstruct on debian/Ubuntu) in order to reconstruct the /var/lib/cyrus/*.db files. I think the main database files you need are mailboxes.db and annotations.db (can someone confirm this?) This still leaves the question of how best to back up a cyrus mailstore. Bron mentioned that most people are using LVM snapshots. I don't see how using btrfs/LVM/ZFS snapshots can save you from a race condition between when the cyrus user directory is updated and when mailboxes.db is updated. The only way I would trust this is by doing this: 1. Stop cyrus 2. Snapshot 3. Restart cyrus cyrdump: near as I can tell the only useful purpose this serves is to assemble all email messages into a single "mbox" file (can anyone confirm this)? ctl_mboxlist: this seems useful for making a human readable copy of the mailboxes.db file, but I'm not sure how this could be useful for disaster recovery, given the previously mentioned issue about keeping the mailboxes.db file synchronized with the contents of the user dir. So, given a simple mail server (i.e. no murder + replication), and when using a filesystem (e.g. ext4 or XFS) which doesn't do snapshots, it would appear that the only safe way to backup up a cyrus mailstore is to either using something like imapsync, or 1. Stop cyrus 2. tar cvf /some/safe/place/user.tar {default-partion} 3. tar cvf /some/safe/place/cyrusdb.tar {configdirectory} 4. Restart cyrus The way I've used imapsync in the past required copying mail folders per authenticated user account; i.e. something like imapsync --host1 my_host1 --authmech1 LOGIN --user1 my_user1 --password1 xxxxx --host2 my_host2 --authmech2 PLAIN --user2 my_user2 --password2 xxxxx which in particular means knowing everyone's passwords. This is entirely unworkable for larger sites, and I'm not sure if there is a trick for getting around this.We are a "large" site. We have 3 backend servers in a Murder cluster with about 25,000 mailboxes per backend. Unless you are going to use Fastmail's fancy backup method that actually locks the mailbox in Cyrus, I don't think there is a way to take a perfectly consistent backup. That said, we didn't care about achieving perfection, so we just perform a normal filesystem backup. We use EMC Networker, but it isn't doing anything fancy. It just walks the entire directory tree looking for changed files to backup. Yeah, there is the potential for a race condition. You really only need each mailbox to be consistent though, and the odds of a mailbox changing while it's being backed up are sufficiently low here. In the event of a disaster (total loss of filesystem), we'll be restoring from our most recent backup, which will be anywhere from 0-24 hours old. If we need to recover an individual mailbox, we can get everything using delayed delete. Either way, we'll be running reconstruct on the mailbox(es). I see people talk a lot about fancy ways to back up Cyrus, but we've just never had a need. Andy
We also don't use snapshots or stop cyrus for backup. But as a complete restore of our mail storage with normal backuptools would take fare too long, we uses cyrus sync for disaster recovery. For the normal backup we use a combination of delayed expung (14 days) and normal filesystem backup. In most cases where the mail is still in the filesystem and can be restored by unexpung, and in the rare cases where the mail has been expunged i have to run reconstruct anyway. Michael -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912Zentrum für Datenverarbeitung mail: michael.menge@xxxxxxxxxxxxxxxxxxxx
Wächterstraße 76 72074 Tübingen
Attachment:
smime.p7s
Description: S/MIME Signatur
---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
