Re: cyrus: delete previously deleted messages(after backup recovery)

[Y B <ybpubs@xxxxxxxxx>]

Thank you for your help, I managed to write a cyrus.index parser that checks all the effective mail UIDs (=filenames) and erases all the unnecessary files (previously deleted and thus not referenced in the index file).

Just in case someone faces the same issue someday, here is the script :
(instead of directly deleting the emails, it creates a bash file with all the rm commands, so it can be reviewed before, just in case something goes wrong...)

In my case, I used the script (cleanMailbox.sh) this way (to clean only the INBOXes) :

for userdir in `find /data/mail/imap/spool/*/user/ -maxdepth 1 -mindepth 1 -name "*"`; do echo $userdir; ~/adminScripts/cleanMailbox.sh "$userdir"; done;

and the script itself:
*************************************************************************************
#!/bin/bash

# cleanMailbox.sh : remove all mailfiles not referenced in the cyrus.index file

if [ ! -d "$1" ]; then
    echo "USAGE: argument must be a valid cyrus user mail directory"
       exit 0
fi

INDEXPATH="$1/cyrus.index"

if [ ! -f "$INDEXPATH" ]; then
    echo "No cyrus.index in $1. ABORT"
    exit 0
fi

#1- read start offset in cyrus.index at position 0x0c ==12 (4 bytes)
STARTOFFSET=`od --skip-bytes=12 --read-bytes=4 -An -t x1 "$INDEXPATH" | tr -d ' '`
STARTOFFSET_10=`printf '%d\n' 0x"$STARTOFFSET"`
echo "messages start offset =0x$STARTOFFSET (=$STARTOFFSET_10)"

#2- read record size in cyrus.index at position 0x10 ==16 (4 bytes)
RECORDSIZE=`od --skip-bytes=16 --read-bytes=4 -An -t x1 "$INDEXPATH" | tr -d ' '`
RECORDSIZE_10=`printf '%d\n' 0x"$RECORDSIZE"`
echo "message record size =0x$RECORDSIZE (=$RECORDSIZE_10)"

#3- read number of emails in cyrus.index at position 0x14 ==20 (4 bytes)
NBMESSAGES=`od --skip-bytes=20 --read-bytes=4 -An -t x1 "$INDEXPATH" | tr -d ' '`
NBMESSAGES_10=`printf '%d\n' 0x"$NBMESSAGES"`
echo "number of emails in metadata=0x$NBMESSAGES (=$NBMESSAGES_10)"

# ALL message UID (=filename) will be in cyrus.index at STARTOFFSET + i * RECORDSIZE
# with i integer in [1..NBMESSAGES]
# Build a reference file to keep track of mail filenames (UID.) that will NOT be deleted
for i in `seq 0 $((NBMESSAGES_10 - 1))`;
  do
    MESSAGEOFFSET=$((STARTOFFSET_10 + i * RECORDSIZE_10))
    UIDMESSAGE=`od --skip-bytes=$MESSAGEOFFSET --read-bytes=4 -An -t x1 "$INDEXPATH" | tr -d ' '`
    UIDMESSAGE_10=`printf '%d\n' 0x"$UIDMESSAGE"`
    echo "$UIDMESSAGE_10". >> "$1"/KEEP_THIS.tmp
done

# instead of deleting directly the undesirable files, build a command file with all the rm commands:
echo cd "$1" >> /tmp/eraseOldMails.sh

cd "$1"
# for all email files
for mailfile in `ls *\.`;
  do
    # if mailfile not found in KEEP_THIS list, delete it:
    grep "^$mailfile$" "$1"/KEEP_THIS.tmp >/dev/null || echo rm "$mailfile" >> /tmp/eraseOldMails.sh
done

chmod +x /tmp/eraseOldMails.sh
rm "$1"/KEEP_THIS.tmp
*************************************************************************************

Yannick Barbeaux

2014-04-23 17:06 GMT+02:00 Michael Menge <michael.menge@xxxxxxxxxxxxxxxxxxxx>:

Hi

Quoting Y B <ybpubs@xxxxxxxxx>:

OK, I managed to reconstruct my test mailbox. The files that were in my
directory structure but not visible in the mailbox are now all visible in
my email client.
Unfortunately, the cyrreconstruct command is not verbose at all and I
cannot see the filenames corresponding to rediscovered emails (so I cannot
delete them...).
Any clue?

...

My version of cyrus : 2.2.13

Cyrus 2.2.13 is very old. AFSIR the -n feature was added in 2.4.x.
Updating in this situation might do harm, as the metadata format
changed in 2.4.x

I think there are no reconstruct options or other tools, to remove the
deleted mails out of the box, but the information which mails where in
the mailbox are stored in the cyrus.index file.
See http://git.cyrusimap.org/cyrus-imapd/tree/doc/internal/mailbox-format.html?h=cyrus-imapd-2.2

So you may be able to write a tool to parse the file.

Regards

    Michael Menge

--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail: michael.menge@xxxxxxx-tuebingen.de
Wächterstraße 76
72074 Tübingen
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus