On Mon, 19 Feb 2007, Gregor Wenkelewsky wrote:
On Thu, 15 Feb 2007 18:15:53 +0100, Andrew Morgan <morgan@xxxxxxxx> wrote:
Cyrus has been installed here just a few weeks ago, and after some hard
days it was working smoothly and very well. Until suddenly, sadly today
it started to fail completely with this error message in mail.warn,
mail.error and syslog:
cyrus/imap[..]: DBERROR: skiplist recovery /var/lib/cyrus/mailboxes.db: 0090 should be ADD or DELETE
cyrus/imap[..]: DBERROR: opening /var/lib/cyrus/mailboxes.db: cyrusdb error
You'll need to fix the corruption of the mailboxes.db file. It is a
skiplist format file in your case, so do a google search for
"skiplist.py". You'll find a python utility that can do some better
recovery than the cyrus tools. The example is for cyrus seen-state files,
but the same should work on the mailboxes.db as well.
Fine, I succeeded with that! Did check the auto backup files before too,
but they were all identical. It was too late probably. Then I used the
skiplist.py from here: http://oss.netfarm.it/python-cyrus.php
python ~/skiplist.py mailboxes.db >mailboxes.txt
rn mailboxes.db mailboxes.err
cvt_cyrusdb /var/lib/cyrus/mailboxes.txt flat /var/lib/cyrus/mailboxes.db skiplist
chown cyrus mailboxes.db
Fixed!
Okay then, now it works, but how often will an error like this occur, can
I do something to prevent it? First I thought of "0090" as some sort of
error code, and I found only two error incidents with /line/ 0090 in
Google... ;) ...but it is much more numerous.
The 0090 is a skiplist offset/index within the file, so the error message
could contain any number depending where the corruption happened.
Can this be related to shutting down and rebooting the system? Could be a
coincidence of course, but just after rebooting the error was there.
Possibly, if Cyrus was stopped (kill -9?) in the middle of a skiplist
operation.
Can it be related to Squatter? By default, Squatter was not set, but some
days before the error I set Squatter to an hourly "nice" run. Now I turned
it off again.
I don't think squatter would have any relation, but I'm not running
squatter here myself.
You should also setup a cronjob to dump the mailboxes.db file to plaintext
periodically (so it can be backed up). Something like this works here:
58 * * * * cyrus /usr/local/cyrus/bin/ctl_mboxlist -d > /var/spool/cyrus/config/mailboxes.db.dump
Yes, I'll do that, though it's more like holding ready the plaster instead
of preventing the injury.
This has to be written to /etc/crontab like Squatter, correct? How often
should it be running? Maybe it's only neccessary when new IMAP users and/or
folders have been created??
Yes, that command above is exactly what I have in my crontab file. I
can't remember why I have it run at 58 minutes after the hour. :)
I feel queasy with an error that has no apparent reason. I wanted to
build a system that can run without administration for months and, maybe,
would sustain even a rare power failure. But there was no power failure
and no sign of a disc error either... :-(
We've been running Cyrus for a couple years now with skiplist for
mailboxes.db. So far we've never had a single corruption of mailboxes.db.
Very rarely we'll get a corrupted username.seen file, which can be fixed
using skiplist.py.
You should be using some sort of journaling filesystem (we use ext3 here)
if you are not already, although that cannot save you from some sources of
skiplist corruption.
Andy
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
