Sometime during the night something happend to my imap store about (23:19 CST). I have executed 'ctl_mboxlist -u < $backup' from two days ago to recover the mboxlist and have done a 'reconstruct -r user.$USER' for a few users to get their $USER.seen folders working again. The process lmtpunix is aborting, I cannot receive email (sendmail to deliver). I can still send mail out (through sendmail) so that part's working.
I'm concerned that I need to recover my primary imap databases.
Can someone help me diagnose what's going on and how to fix it (and what happened)?
I am a small site with less than 30 email accounts in cyrus.
There were no cyrus, sendmail, nor other email changes yesterday.
Mike
Fedora Core 5 (current patches)
$ rpm -qa | grep cyrus
cyrus-imapd-2.3.1-2.8.fc5
cyrus-imapd-devel-2.3.1-2.8.fc5
cyrus-imapd-utils-2.3.1-2.8.fc5
cyrus-sasl-2.1.21-10
cyrus-sasl-devel-2.1.21-10
cyrus-sasl-ldap-2.1.21-10
cyrus-sasl-lib-2.1.21-10
$ rpm -qa | grep db
db4-4.3.29-8.fc5
db4-devel-4.3.29-8.fc5
db4-utils-4.3.29-8.fc5
dbus-0.62-0.2.fc5
dbus-devel-0.62-0.2.fc5
dbus-glib-0.62-0.2.fc5
dbus-python-0.62-0.2.fc5
dbus-sharp-0.62-0.2.fc5
dbus-x11-0.62-0.2.fc5
dhcdbd-1.15-2.FC5
gdb-6.3.0.0-1.134.fc5
gdbm-1.8.0-26.2
gdbm-devel-1.8.0-26.2
jakarta-commons-dbcp-1.2.1-3jpp_3fc
libdbi-0.8.1-1.2.1
libdbi-dbd-mysql-0.8.1a-1.2.1
libdbi-drivers-0.8.1a-1.2.1
mysql-connector-odbc-3.51.12-1.2.1
nss_db-2.2-35
entries from /var/log/maillog
Mar 26 08:59:23 elo master[20695]: service lmtpunix pid 23909 in READY state: terminated abnormally
Mar 26 08:59:23 elo master[23979]: about to exec /usr/lib/cyrus-imapd/lmtpd
Mar 26 08:59:23 elo lmtpunix[23979]: executed
Mar 26 08:59:24 elo sendmail[22334]: m2QDfxl0022334: ruleset=check_mail, arg1=<hlmbpzwby@xxxxxxxxxxxxxxxxxxxxx>, relay=localhost [127.0.
0.1], reject=451 4.1.8 Domain of sender address hlmbpzwby@xxxxxxxxxxxxxxxxxxxxx does not resolve
Mar 26 08:59:24 elo sm-msp-queue[21753]: m2LNUjr6004989: to=$USER@$DOMAIN, delay=4+14:28:39, xdelay=00:00:02, mailer=relay, pri=97
52409, relay=[127.0.0.1], dsn=4.0.0, stat=Deferred: 451 4.1.8 Domain of sender address hlmbpzwby@xxxxxxxxxxxxxxxxxxxxx does not resolve
Mar 26 08:59:25 elo pop3[23212]: accepted connection
Mar 26 08:59:25 elo master[23980]: about to exec /usr/lib/cyrus-imapd/pop3d
Mar 26 08:59:25 elo pop3[23980]: executed
Mar 26 08:59:25 elo pop3[23212]: login: $MACHINE.$DOMAIN [10.1.2.77] $USER plaintext User logged in
Mar 26 08:59:29 elo imap[20839]: SQUAT failed to open index file
Mar 26 08:59:29 elo imap[20839]: SQUAT failed
Mar 26 08:59:29 elo sendmail[23958]: m2QDxBv3023958: from=<IOR@xxxxxxxxxxxxx>, size=6096, class=0, nrcpts=1, msgid=<E1JeW9V-00018C-OB@ce
valvorto41.vauntstrand.com>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:30 elo sendmail[23984]: m2QDxBv3023958: to=<$USER@$DOMAIN>, delay=00:00:18, xdelay=00:00:00, mailer=cyrusv2, pri=126
096, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:32 elo master[20695]: process 23955 exited, signaled to death by 7
Mar 26 08:59:32 elo master[20695]: service lmtpunix pid 23955 in READY state: terminated abnormally
Mar 26 08:59:32 elo master[23988]: about to exec /usr/lib/cyrus-imapd/lmtpd
Mar 26 08:59:32 elo lmtpunix[23988]: executed
Mar 26 08:59:32 elo sendmail[23957]: m2QDxACR023957: from=<uliedtke@xxxxxxxxxxxx>, size=8641, class=0, nrcpts=1, msgid=<151501c88f49$94a
d7780$7c475ec8@Nichole>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:32 elo sendmail[23959]: m2QDxDQY023959: from=<NoelleselectmanGoldsmith@xxxxxxxxxxxxxxx>, size=6314, class=0, nrcpts=1, msgi
d=11fa701c88f49$97279f40$57e39cca@SN2564177124, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:33 elo sendmail[23990]: m2QDxACR023957: to=<fmpcaqfc@$DOMAIN>, delay=00:00:19, xdelay=00:00:00, mailer=cyrusv2, pri=1
28641, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:33 elo sendmail[23992]: m2QDxDQY023959: to=<$USER@$DOMAIN>, delay=00:00:18, xdelay=00:00:00, mailer=cyrusv2, pri=1263
14, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:33 elo sendmail[23938]: m2QDwxuZ023938: from=<n@xxxxxxxxxxxxx>, size=11014, class=0, nrcpts=1, msgid=<000601c88f49$04b7aef1
$413589ac@hmwrugyt>, proto=ESMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:34 elo sendmail[23994]: m2QDwxuZ023938: to=<$USER@$DOMAIN>, delay=00:00:27, xdelay=00:00:00, mailer=cyrusv2, pri=13
1014, relay=localhost, dsn=4.4.2, stat=Deferred: Connection timed out with localhost
Mar 26 08:59:40 elo sendmail[23967]: m2QDxJCu023967: from=<uliedtke@xxxxxxxxxxxx>, size=7569, class=0, nrcpts=1, msgid=<154301c88f49$98c
41130$7c475ec8@Nichole>, proto=SMTP, daemon=MTA, relay=$DMZ.$DOMAIN [10.1.2.1]
Mar 26 08:59:40 elo master[20695]: process 23916 exited, signaled to death by 7
Mar 26 08:59:40 elo master[20695]: service lmtpunix pid 23916 in READY state: terminated abnormally
Mar 26 08:59:40 elo master[23998]: about to exec /usr/lib/cyrus-imapd/lmtpd
Mar 26 08:59:40 elo lmtpunix[23998]: executed
Mar 26 08:59:40 elo sendmail[23997]: m2QDxJCu023967: to=$USER, delay=00:00:19, xdelay=00:00:00, mailer=cyrusv2, pri=37785, relay=local
$ db_verify deliver.db 2>&1 | head
db_verify: deliver.db: unexpected file type or format
db_verify: Page 0: pgno incorrectly set to 1953720684
db_verify: Page 0: bad magic number 543582572
db_verify: Page 0: bad page size 16777216
db_verify: Page 1: bad page number 1206496269
db_verify: Page 1: bad page type 255
db_verify: Page 1: unknown page type 255
db_verify: Page 2: bad page number 1701145449
db_verify: Page 2: bad page type 142
db_verify: Page 2: unknown page type 142
......
db_verify: Page 228: bad page type 109
db_verify: Page 228: unknown page type 109
db_verify: Page 229: bad page number 1465276490
db_verify: Page 229: bad page type 105
db_verify: Page 229: unknown page type 105
db_verify: deliver.db: DB_VERIFY_BAD: Database verification failed
$ file *.db
annotations.db: Cyrus skiplist DB
deliver.db: Cyrus skiplist DB
mailboxes.db: Cyrus skiplist DB
tls_sessions.db: Cyrus skiplist DB
$ file *.db
annotations.db: Cyrus skiplist DB
deliver.db: Cyrus skiplist DB
mailboxes.db: Cyrus skiplist DB
tls_sessions.db: Cyrus skiplist DB
[cyrus@elo ~]$ cat /etc/cyrus.conf
# standard standalone server implementation
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
idled cmd="idled"
}
# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
# add or remove based on preferences
imap cmd="imapd" listen="imap" prefork=10
imaps cmd="imapd -s" listen="imaps" prefork=5
pop3 cmd="pop3d" listen="pop3" prefork=2
pop3s cmd="pop3d -s" listen="pop3s" prefork=2
sieve cmd="timsieved" listen="sieve" prefork=10
# these are only necessary if receiving/exporting usenet via NNTP
# nntp cmd="nntpd" listen="nntp" prefork=3
# nntps cmd="nntpd -s" listen="nntps" prefork=1
# at least one LMTP is required for delivery
# lmtp cmd="lmtpd" listen="localhost:lmtp" prefork=1
lmtpunix cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=10
# this is only necessary if using notifications
# notify cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp" prefork=1
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if using duplicate delivery suppression,
# Sieve or NNTP
delprune cmd="cyr_expire -E 3 -X 1" at=0200
# this is only necessary if caching TLS sessions
tlsprune cmd="tls_prune" at=0400
# mikee 080109 - make sure messages are trimmed
msgprune1 cmd="ipurge -f -d 2 -X user.%.backup" at=2200
msgprune2 cmd="ipurge -f -d 5 -X user.%.spam" at=2230
msgprune3 cmd="ipurge -f -d 4 -X user.%.Trash" at=2300
}
$ cat /etc/imapd.conf
admins: cyrus
allowanonymouslogin: no
allowplaintext: yes
autocreatequota: -1
autocreateinboxfolders: Sent | Drafts | | spam | backup
configdirectory: /var/lib/imap
drachost: localhost
dracinterval: 0
hashimapspool: true
partition-default: /var/spool/imap
poptimeout: 10
postmaster: postmaster
quotawarn: 90
reject8bit: no
sasl_mech_list: PLAIN
sasl_pwcheck_method: saslauthd
sendmail: /usr/lib/sendmail
servername: servername
sieveusehomedir: 0
sievedir: /var/lib/imap/sieve
sieve_maxscripts: 5
sieve_maxscriptsize: 96
timeout: 1440
tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd-071024.pem
tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd-071024.pem
#unixhierarchysep: yes
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
