Re: copying and renaming mailboxes to clean up directories

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, 2 May 2007, Thorsten Büker wrote:

Dear all,

For quite a few month I ran a backup-scenario set up according to [1]
including synchronisation of the mailboxes using "rsync -vaR". (In the
meantime I migrated to the usage of LVM snapshots.)

As restoring a backup became necessary due to filesystem corruption I
fetched the mailboxes from the r'synched folders as well as the mailbox databases and further necessary stuff. The restore went smoothly.

But wondering about the heavily increased amount of partition in use, I
noticed that my call of rsync lacked of the "--delete" attribute. In consequence the restored mailbox directories include not only the messages, which should be in the inbox, but all files beeing in the directories during the backup cronjobs. (My mailbox sizes ~700 MB instead of ~100 MB.)

With regard to backup performance I'd like to clean up the relevant
directories by ensuring uniformity of the inbox according to the Cyrus
mailbox database and the files in the mailbox directory.

In the wiki I found Andrew's collection of scripts [2] and had a try. My intention is/was to firstly manually select mailbox directories, which contain much too much unneeded files -- mainly mailboxes used via imap. After temporarily stopping mail delivery and mailbox access, each chosen mailbox might be renamed and its contents copied to a new mailbox of its initial name. After starting mail delivery and access again, the mailboxes and directories containing wrong fails might be deleted at all.

But, as usual, the idea doesn't work yet. A first try to copy the
existing mailbox "thorsten" to "thorsten_neu" leads to the following:

~~~~

mail:/var/spool/cyrus/mail/t/user#
/usr/local/sbin/cyrus/copy_user_mailbox.pl thorsten thorsten_neu
Successfully created inbox for thorsten_neu
Successfully set quota for thorsten_neu to 102400
"user.thorsten"
Copying folder user.thorsten to user.thorsten_neu
"user.thorsten.Archiv"
Copying folder user.thorsten.Archiv to user.thorsten_neu.Archiv
"user.thorsten.Drafts"
Copying folder user.thorsten.Drafts to user.thorsten_neu.Drafts
"user.thorsten.Mailinglisten"
Copying folder user.thorsten.Mailinglisten to
user.thorsten_neu.Mailinglisten
Error copying messages to user.thorsten_neu.Mailinglisten
"user.thorsten.Mailinglisten.Announces"
Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
Error sending '24 SELECT user.thorsten.Mailinglisten.Announces' to IMAP:
at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
Copying folder user.thorsten.Mailinglisten.Announces to
user.thorsten_neu.Mailinglisten.Announces
Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
Error sending '25 STATUS user.thorsten.Mailinglisten (MESSAGES)' to
IMAP:  at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
Use of uninitialized value in numeric eq (==) at
/usr/local/sbin/cyrus/copy_user_mailbox.pl line 129.

[.. various errors at lines 104 and 129 ..]

Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 152
Error sending '84 LOGOUT' to IMAP:  at
/usr/local/sbin/cyrus/copy_user_mailbox.pl line 152

~~~~

Please note, the additional output on the folder name results from
uncommenting line 92 of the script. The relevant sections look like
below [6]. Evaluating, why the connection to Cyrus gets lost while looping through the folders, a discovered "word too long" in /var/log/mail.log.

Well, in this case (my personal inbox) erasing various mails inside "user.thorsten.Mailinglisten" (~2200 mails) helped to make copying work. But regarding customer's inboxes this might not be first choice ;-)

In google I found [3], suggesting to recompile imapd. Alas, is there any way to modify the script instead of recompiling imapd to permit a larger value? Indeed I'd prefer to keep sarge's "2.1.18-1+sarge2".

Ack, someone using my scripts!  :)

I bet the problem is that on a large mailbox, $imap->messages will return a really big list of UIDs (longer than the allowed MAXWORD of 32768). You may be able to fix this by calling "$imap->Ranges(1);" earlier in the script. According to the Mail::IMAPClient docs, with that enabled it will try to generate a condensed list of UIDs when possible.

However, if that still does not work then you may have to add a loop to the code to operate on "chunks" of UIDs at a time, rather than the whole list at once. Instead of:

        my $uidlist = $imap1->copy($targetf, $imap1->messages);

you'd do something like:

	@alluids = $imap1->messages;
	foreach $uid (@alluids) {
		$uiddone = $imap1->copy($targetf, $uid);
	}

There is probably some smarter way to use bigger chunks of uids rather than 1 at a time.

Furthermore I tried to rename a mailbox via cyradm in advance, but this leads to the following:

~~~~

mail.myhostname.de> renm user.thorsten user.thorsten_orig
renamemailbox: Operation is not supported on mailbox

~~~~

Did you set "allowusermoves: 1" in imapd.conf?

The amount of results by google is rather poor, honestly spoken one result [4]. The presented workaround [5] contains quite a lot of work dealing manually with each subfolder. Is there any other way to rename a mailbox?

Or, going even further, is there any more clever attempt than the one above to solve the problem? Thanks in advance!

If I understand you correctly, you want to get a list of "valid" UIDs from the folder listing, and then delete (from the filesystem) anything which is not in that list of "valid" UIDs?

Or do you intend to just copy the mailbox to a new mailbox, delete the old mailbox, and copy it back? That sounds pretty reasonable to me. One problem you might have is the loss of the seen state for each mailbox because the mailbox unique ID will change with the copy. The unique ID is stored in the cyrus.header file in each mailbox/folder, so you might be able to save those and copy them back as you go.

An ugly problem!

	Andy
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

[Index of Archives]     [Cyrus SASL]     [Squirrel Mail]     [Asterisk PBX]     [Video For Linux]     [Photo]     [Yosemite News]     [gtk]     [KDE]     [Gimp on Windows]     [Steve's Art]

  Powered by Linux