On Wed, 2 May 2007, Thorsten Büker wrote:
Dear all,
For quite a few month I ran a backup-scenario set up according to [1]
including synchronisation of the mailboxes using "rsync -vaR". (In the
meantime I migrated to the usage of LVM snapshots.)
As restoring a backup became necessary due to filesystem corruption I
fetched the mailboxes from the r'synched folders as well as the mailbox
databases and further necessary stuff. The restore went smoothly.
But wondering about the heavily increased amount of partition in use, I
noticed that my call of rsync lacked of the "--delete" attribute. In
consequence the restored mailbox directories include not only the messages,
which should be in the inbox, but all files beeing in the directories during
the backup cronjobs. (My mailbox sizes ~700 MB instead of ~100 MB.)
With regard to backup performance I'd like to clean up the relevant
directories by ensuring uniformity of the inbox according to the Cyrus
mailbox database and the files in the mailbox directory.
In the wiki I found Andrew's collection of scripts [2] and had a try. My
intention is/was to firstly manually select mailbox directories, which
contain much too much unneeded files -- mainly mailboxes used via imap. After
temporarily stopping mail delivery and mailbox access, each chosen mailbox
might be renamed and its contents copied to a new mailbox of its initial
name. After starting mail delivery and access again, the mailboxes and
directories containing wrong fails might be deleted at all.
But, as usual, the idea doesn't work yet. A first try to copy the
existing mailbox "thorsten" to "thorsten_neu" leads to the following:
~~~~
mail:/var/spool/cyrus/mail/t/user#
/usr/local/sbin/cyrus/copy_user_mailbox.pl thorsten thorsten_neu
Successfully created inbox for thorsten_neu
Successfully set quota for thorsten_neu to 102400
"user.thorsten"
Copying folder user.thorsten to user.thorsten_neu
"user.thorsten.Archiv"
Copying folder user.thorsten.Archiv to user.thorsten_neu.Archiv
"user.thorsten.Drafts"
Copying folder user.thorsten.Drafts to user.thorsten_neu.Drafts
"user.thorsten.Mailinglisten"
Copying folder user.thorsten.Mailinglisten to
user.thorsten_neu.Mailinglisten
Error copying messages to user.thorsten_neu.Mailinglisten
"user.thorsten.Mailinglisten.Announces"
Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
Error sending '24 SELECT user.thorsten.Mailinglisten.Announces' to IMAP:
at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
Copying folder user.thorsten.Mailinglisten.Announces to
user.thorsten_neu.Mailinglisten.Announces
Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
Error sending '25 STATUS user.thorsten.Mailinglisten (MESSAGES)' to
IMAP: at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
Use of uninitialized value in numeric eq (==) at
/usr/local/sbin/cyrus/copy_user_mailbox.pl line 129.
[.. various errors at lines 104 and 129 ..]
Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 152
Error sending '84 LOGOUT' to IMAP: at
/usr/local/sbin/cyrus/copy_user_mailbox.pl line 152
~~~~
Please note, the additional output on the folder name results from
uncommenting line 92 of the script. The relevant sections look like
below [6]. Evaluating, why the connection to Cyrus gets lost while looping
through the folders, a discovered "word too long" in /var/log/mail.log.
Well, in this case (my personal inbox) erasing various mails inside
"user.thorsten.Mailinglisten" (~2200 mails) helped to make copying work. But
regarding customer's inboxes this might not be first choice ;-)
In google I found [3], suggesting to recompile imapd. Alas, is there any way
to modify the script instead of recompiling imapd to permit a larger value?
Indeed I'd prefer to keep sarge's "2.1.18-1+sarge2".
Ack, someone using my scripts! :)
I bet the problem is that on a large mailbox, $imap->messages will return
a really big list of UIDs (longer than the allowed MAXWORD of 32768). You
may be able to fix this by calling "$imap->Ranges(1);" earlier in the
script. According to the Mail::IMAPClient docs, with that enabled it will
try to generate a condensed list of UIDs when possible.
However, if that still does not work then you may have to add a loop to
the code to operate on "chunks" of UIDs at a time, rather than the whole
list at once. Instead of:
my $uidlist = $imap1->copy($targetf, $imap1->messages);
you'd do something like:
@alluids = $imap1->messages;
foreach $uid (@alluids) {
$uiddone = $imap1->copy($targetf, $uid);
}
There is probably some smarter way to use bigger chunks of uids rather
than 1 at a time.
Furthermore I tried to rename a mailbox via cyradm in advance, but this leads
to the following:
~~~~
mail.myhostname.de> renm user.thorsten user.thorsten_orig
renamemailbox: Operation is not supported on mailbox
~~~~
Did you set "allowusermoves: 1" in imapd.conf?
The amount of results by google is rather poor, honestly spoken one result
[4]. The presented workaround [5] contains quite a lot of work dealing
manually with each subfolder. Is there any other way to rename a mailbox?
Or, going even further, is there any more clever attempt than the one above
to solve the problem? Thanks in advance!
If I understand you correctly, you want to get a list of "valid" UIDs from
the folder listing, and then delete (from the filesystem) anything which
is not in that list of "valid" UIDs?
Or do you intend to just copy the mailbox to a new mailbox, delete the old
mailbox, and copy it back? That sounds pretty reasonable to me. One
problem you might have is the loss of the seen state for each mailbox
because the mailbox unique ID will change with the copy. The unique ID is
stored in the cyrus.header file in each mailbox/folder, so you might be
able to save those and copy them back as you go.
An ugly problem!
Andy
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
