Replication errors in 2.3.16-8 (do_user bailing out)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I'm running into inconsistent errors from sync_client on a Cyrus 2.3.16-8 system.  It's driving me crazy because the same sync_client command will succeed at some times and fail at others.  I know others are running similar and much more complex systems successfully, so I must be missing something simple or fundamental.

Thanks in advance,

John

When the problem occurs, sync_client gives a "do_client(user@xxxxxxxxxxx): bailing out!" error and logs a message about failing to reset the replication (cyrusadmin) account.  This occurs both when running sync_client against a single user and when using sync_client with a file containing 27,000 user names.  The failures do not always effect the same users, and attempts to replicate a given user generally succeed after 1-5 tries.  When running the 27,000 user test the failure can occur for anywhere from a handful to a few hundred users.  The problem occurs whether sync_client is run by root, by root "su -c"' to cyrus, or as the cyrus user.  Running "reconstruct", "reconstruct -G", or "reconstruct -x -f" prior to running sync_client does not resolve the problem.  

In addition to the "do_client(user@xxxxxxxxxxx): bailing out!" failures, users are sometimes skipped without a  sync_client error message -- they simply aren't replicated on that pass.  Sync_client displays the line "USER user@xxxxxxxxxxx" but none of the ADDSUB lines, similar to when processing a user that has never used their mailbox.

Both problems occur whether or not rolling replication is running.

Both the master mailstore (eml-store04) and the replica (eml-replica04) are CentOS 5.4 64-bit servers running as virtual machines on a single ESXi 4 server.  They are using the ext3 filesystem.  The cyrus-imapd and cyrus-imapd-utils RPMS were built from the Invoca RPMS on a similar system.  I enabled SNMP support, but didn't make any other changes to the spec file before running "rpmbuild -ba cyrus-imapd.spec".  No errors were reported during the build.

The data was transferred from a Cyrus 2.3.7 system using rsync on /var/lib/imap and /var/spool/imap, and appears normal from a Cyrus and IMAP client perspective.  Between tests, I clear the replica with "rm -rf /var/lib/imap/domain/* /var/lib/imap/sieve/* /var/spool/imap/domain/*".  Everything under /var/lib/imap and /var/spool/imap is owned by cyrus:mail.

[root@eml-store04 ~]# uname -a
Linux eml-store04 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9 12:54:20 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@eml-replica04 reyrey.net]# rpm -qa | grep cyrus | sort
cyrus-imapd-2.3.16-8
cyrus-imapd-utils-2.3.16-8
cyrus-sasl-2.1.22-5.el5_4.3
cyrus-sasl-lib-2.1.22-5.el5_4.3
cyrus-sasl-lib-2.1.22-5.el5_4.3
cyrus-sasl-plain-2.1.22-5.el5_4.3
cyrus-sasl-plain-2.1.22-5.el5_4.3

Failed attempt:
[root@eml-store04 ~]# /usr/lib/cyrus-imapd/sync_client -l -v -u -f mailboxlist.ldap

eml-store04: output from sync_client:
USER admin01@xxxxxxxxxxx
Error from do_user(admin01@xxxxxxxxxxx): bailing out!

eml-store04: content of /var/log/maillog
Nov 18 11:37:27 eml-store04 sync_client[6328]: USER admin01@xxxxxxxxxxx
Nov 18 11:37:27 eml-store04 sync_client[6328]: USER received NO response: IMAP_MAILBOX_NONEXISTENT Failed to access inbox for admin01@xxxxxxxxxxx: System I/O error
Nov 18 11:37:27 eml-store04 sync_client[6328]: RESET received NO response: Failed to reset account cyrusadmin: Internal Error
Nov 18 11:37:27 eml-store04 sync_client[6328]: Error in do_user(admin01@xxxxxxxxxxx): bailing out!

eml-replica04: content of /var/log/maillog
Nov 18 11:37:27 eml-replica04 syncserver[12635]: IOERROR: opening /var/spool/imap/domain/r/reptest.org/a/user/admin01/cyrus.header: No such file or directory
Nov 18 11:37:27 eml-replica04 syncserver[12635]: Failed to access inbox for admin01@xxxxxxxxxxx
Nov 18 11:37:27 eml-replica04 syncserver[12635]: IOERROR: opening /var/spool/imap/domain/r/reptest.org/a/user/admin01/cyrus.header: No such file or directory
Nov 18 11:37:27 eml-replica04 syncserver[12635]: Unlocked


Successful attempt:
[root@eml-store04 ~]# /usr/lib/cyrus-imapd/sync_client -l -v -u admin01@xxxxxxxxxxx
eml-store04:output from sync_client:
USER admin01@xxxxxxxxxxx
ADDSUB admin01@xxxxxxxxxxx INBOX
ADDSUB admin01@xxxxxxxxxxx INBOX.Drafts
ADDSUB admin01@xxxxxxxxxxx INBOX.Sent Items
ADDSUB admin01@xxxxxxxxxxx INBOX.Trash

eml-store04: content of /var/log/maillog:
Nov 18 11:54:47 eml-store04 sync_client[6416]: USER admin01@xxxxxxxxxxx
Nov 18 11:54:47 eml-store04 sync_client[6416]: USER received NO response: IMAP_MAILBOX_NONEXISTENT Failed to access inbox for admin01@xxxxxxxxxxx: Mailbox does not exist
Nov 18 11:54:48 eml-store04 sync_client[6416]: ADDSUB admin01@xxxxxxxxxxx INBOX
Nov 18 11:54:48 eml-store04 sync_client[6416]: ADDSUB admin01@xxxxxxxxxxx INBOX.Drafts
Nov 18 11:54:48 eml-store04 sync_client[6416]: ADDSUB admin01@xxxxxxxxxxx INBOX.Sent Items
Nov 18 11:54:48 eml-store04 sync_client[6416]: ADDSUB admin01@xxxxxxxxxxx INBOX.Trash

eml-replica04: content of /var/log/maillog
Nov 18 11:54:47 eml-replica04 syncserver[14261]: executed
Nov 18 11:54:47 eml-replica04 syncserver[14261]: accepted connection
Nov 18 11:54:47 eml-replica04 syncserver[14261]: cmdloop(): startup
Nov 18 11:54:47 eml-replica04 syncserver[14261]: login: eml-store04.asddev.reyrey.com [172.29.97.2] cyrusadmin PLAIN User logged in
Nov 18 11:54:47 eml-replica04 syncserver[14261]: Failed to access inbox for admin01@xxxxxxxxxxx
Nov 18 11:54:47 eml-replica04 syncserver[14261]: cannot unlink /var/lib/imap/domain/r/reptest.org/user/a/admin01.seen: No such file or directory
Nov 18 11:54:47 eml-replica04 syncserver[14261]: cannot unlink /var/lib/imap/domain/r/reptest.org/user/a/admin01.mboxkey: No such file or directory
Nov 18 11:54:48 eml-replica04 syncserver[14261]: Unlocked

[root@eml-store04 ~]# head mailboxlist.ldap
Service.Administrator@xxxxxxxxxxxxxxxxx
postmaster@xxxxxxxxxx
test@xxxxxxxxxx
test1@xxxxxxxxxx
bill@xxxxxxxxxx
appproxy@xxxxxxxxxx
abuse@xxxxxxxxxx
adminlist@xxxxxxxxxx

John Simpson 
Senior Software Engineer, I. T. Engineering and Operations

Attachment: eml-store04-imapd.conf
Description: eml-store04-imapd.conf

Attachment: eml-replica04-cyrus.conf
Description: eml-replica04-cyrus.conf

Attachment: eml-replica04-imapd.conf
Description: eml-replica04-imapd.conf

Attachment: eml-store04-cyrus.conf
Description: eml-store04-cyrus.conf

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

[Index of Archives]     [Cyrus SASL]     [Squirrel Mail]     [Asterisk PBX]     [Video For Linux]     [Photo]     [Yosemite News]     [gtk]     [KDE]     [Gimp on Windows]     [Steve's Art]

  Powered by Linux