A few corrections of typos..
On 05/19/2017 05:48 PM, Michael Hieb wrote:
Apologies for long post, most of it is configuration details.
I have recently changed from a single threaded cyrus server in which
multiple domains were overloaded on one listener as multiple virtual
domains to a multiple threaded cyrus server in which multiple domains
are listened for on separate ip addresses with separate
configurations. The primary motivation was to have separate tls
certificates for each domain. The problem is that I can connect to
cyrus listener on all ip/ports for all domains, but while one of them
will fork immediately and respond with a banner, the others will delay
for a period of time from a few seconds to a minute or so before
forking and responding with a banner. With the same setup, I change to
a single threaded cyrus server to listen on all ip/ports then I get an
immediate response on all, but of course I do not get the separate tls
certificate or configuration. It is replicable that switching between
the single threaded and multiple threaded configuration triggers the
problem. I have searched the logs and the mail-lists and found nothing
that seems related.
Question: why does switching to multiple threaded cyrus server trigger
delayed forking of imapd processes after connecting to master listener?
On listener which responds immediately (as expected) I get this:
user@somehost:~> telnet imap.domain1.com 143
Trying 192.168.110.171...
Connected to imap.domain1.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=GSS-SPNEGO
AUTH=GSSAPI AUTH=DIGEST-MD5 AUTH=CRAM-MD5 AUTH=LOGIN AUTH=PLAIN
SASL-IR] imap.domain1.com Cyrus IMAP v2.4.18 server ready
On listener with delay, I get this (and process hangs for many seconds
or even a minute or two):
user@somehost:~> telnet imap.domain2.com 143
Trying 192.168.110.172...
Connected to imap.domain2.com.
Escape character is '^]'.
The domain for which the listener responds and the one for which it
delays seems to change and be random as far as I can tell.
There is nothing in the log journal that looks different from when we
connect and get a for and when we connect and get a delay.
May 19 09:44:31 MAILSERVER master[13762]: about to exec
/usr/lib/cyrus/bin/imapd
May 19 09:44:31 MAILSERVER imap[13762]: executed
May 19 09:44:31 MAILSERVER imap[13762]: IOERROR: opening
/var/lib/imap/user_deny.db: No such file or directory
and once imapd forks and banner is generated
May 19 09:46:45 MAILSERVER imap[13762]: accepted connection
Here are the configuration details:
I run cyrus 2.4.18-3.6 on openSuSE Leap 42.2 Linux MAILSERVER
4.4.62-18.6-default #1 SMP Fri Apr 21 16:14:48 UTC 2017 (84f9824)
x86_64 x86_64 x86_64 GNU/Linux.
Here is my (sanitized) cyrus.conf and one imapd.conf (they all look
alike except for certificate and domain specifics).
MAILSERVER:~ # cat /etc/cyrus.conf
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
idled cmd="idled"
}
# UNIX sockets start with a slash and are put into /var/lib/imap/socket
SERVICES {
# add or remove based on preferences
#imap cmd="imapd" listen="imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain1.com.conf "
listen="192.168.171.4:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain1.com.conf "
listen="192.168.110.171:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain2.com.conf "
listen="192.168.172.4:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain2.com.conf "
listen="192.168.110.172:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain3.com.conf "
listen="192.168.174.4:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain3.com.conf "
listen="192.168.110.174:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain4.com.conf "
listen="192.168.175.4:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain4.com.conf "
listen="192.168.110.175:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain5.com.conf "
listen="192.168.176.4:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain5.com.conf "
listen="192.168.110.176:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain6.com.conf "
listen="192.168.177.4:imap" maxchild=-1 maxforkrate=100
imap cmd="imapd -C /etc/imapd.domain6.com.conf "
listen="192.168.110.177:imap" maxchild=-1 maxforkrate=100
#imaps cmd="imapd -s" listen="imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain1.com.conf -s "
listen="192.168.171.4:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain1.com.conf -s "
listen="192.168.110.171:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain2.com.conf -s "
listen="192.168.172.4:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain2.com.conf -s "
listen="192.168.110.172:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain3.com.conf -s "
listen="192.168.174.4:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain3.com.conf -s "
listen="192.168.110.174:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain4.com.conf -s "
listen="192.168.175.4:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain4.com.conf -s "
listen="192.168.110.175:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain5.com.conf -s "
listen="192.168.176.4:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain5.com.conf -s "
listen="192.168.110.176:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain6.com.conf -s "
listen="192.168.177.4:imaps" maxchild=-1 maxforkrate=100
imaps cmd="imapd -C /etc/imapd.domain6.com.conf -s "
listen="192.168.110.177:imaps" maxchild=-1 maxforkrate=100
#pop3 cmd="pop3d" listen="pop3"
#pop3s cmd="pop3d -s" listen="pop3s"
#sieve cmd="timsieved" listen="sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain1.com.conf"
listen="192.168.171.4:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain1.com.conf"
listen="192.168.110.171:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain2.com.conf"
listen="192.168.172.4:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain2.com.conf"
listen="192.168.110.172:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain3.com.conf"
listen="192.168.174.4:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain3.com.conf"
listen="192.168.110.174:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain4.com.conf"
listen="192.168.175.4:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain4.com.conf"
listen="192.168.110.175:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain5.com.conf"
listen="192.168.176.4:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain5.com.conf"
listen="192.168.110.176:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain6.com.conf"
listen="192.168.177.4:sieve" maxchild=-1 maxforkrate=100
sieve cmd="timsieved -C /etc/imapd.domain6.com.conf"
listen="192.168.110.177:sieve" maxchild=-1 maxforkrate=100
#ptloader cmd="ptloader" listen="/var/lib/imap/ptclient/ptsock"
# at least one LMTP is required for delivery
lmtp cmd="lmtpd -a" listen="mail.domain1.com:lmtp" maxchild=-1
maxforkrate=100
lmtp cmd="lmtpd -a" listen="mail.domain2.com:lmtp" maxchild=-1
maxforkrate=100
lmtp cmd="lmtpd -a" listen="mail.domain3.com:lmtp" maxchild=-1
maxforkrate=100
lmtp cmd="lmtpd -a" listen="mail.domain4.com:lmtp" maxchild=-1
maxforkrate=100
lmtp cmd="lmtpd -a" listen="mail.domain5.com:lmtp" maxchild=-1
maxforkrate=100
lmtp cmd="lmtpd -a" listen="mail.domain6.com:lmtp" maxchild=-1
maxforkrate=100
#lmtpunix cmd="lmtpd" listen="/var/lib/imap/socket/lmtp"
# this is only necessary if using notifications
#notify cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp"
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if using duplicate delivery suppression,
# Sieve or NNTP
duplicateprune cmd="cyr_expire -E 3" at=0400
# Expire data older then 69 days. Two full months of 31 days
# each includes two full backup cycles, plus 1 week margin
# because we run our full backups on the first sat/sun night
# of each month.
deleteprune cmd="cyr_expire -E 4 -D 69" at=0430
expungeprune cmd="cyr_expire -E 4 -X 69" at=0445
# this is only necessary if caching TLS sessions
tlsprune cmd="tls_prune" at=0400
# Uncomment the next entry, if you want to automatically remove
# old messages of EVERY user.
# This example calls ipurge every 60 minutes and ipurge will delete
# ALL messages older then 120 days.
# enter 'man 8 ipurge' for more details
#cleanup cmd="ipurge -d 120 -f" period=60
cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain1.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain2.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain3.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain4.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain5.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Spam*@domain6.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain1.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain2.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain3.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain4.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain5.com" period=60
cleanup cmd="ipurge -f -d 30 user/%/Trash*@domain6.com" period=60
cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain1.com" period=60
cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain2.com" period=60
cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain3.com" period=60
cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain4.com" period=60
cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain5.com" period=60
cleanup cmd="ipurge -f -d 60 user/%/Sent*@domain6.com" period=60
# Create search indexes regularly
squatter cmd="squatter -s -i" at=0530
# running sa-learn
sa-learn cmd="/usr/local/bin/cyrus-salearn.pl" period=60
sa-update cmd="/usr/bin/sa-update -v" at=0000
}
MAILSERVER:~ # cat /etc/imapd.domain1.com.conf
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
sievedir: /var/lib/sieve
annotation_definitions: /etc/imapd.annotations.conf
# admins: cyrus@xxxxxxxxxxx
allowanonymouslogin: no
autocreatequota: 10000
reject8bit: no
quotawarn: 90
timeout: 30
poptimeout: 10
dracinterval: 0
drachost: localhost
sasl_pwcheck_method: saslauthd
lmtp_overquota_perm_failure: no
#lmtp_catchall_mailbox: admin
lmtp_downcase_rcpt: yes
lmtp_fuzzy_mailbox_match: yes
expunge_mode: delayed
deletedprefix: DELETED
delete_mode: delayed
# added by Michael Hieb Jun 22 2014
allowplaintext: yes
unixhierarchysep: yes
allowplainwithouttls: no
altnamespace: no
virtdomains: userid
servername: imap.domain1.net
#
# if you want TLS, you have to generate certificates and keys
#
tls_cert_file: /etc/letsencrypt/live/imap.domain1.net/cert.pem
tls_key_file: /etc/letsencrypt/live/imap.domain1.net/privkey.pem
tls_ca_file: /etc/letsencrypt/live/imap.domain1.net/chain.pem
tls_ca_path: /etc/ssl/certs
tls_versions: tls1_0 tls1_1 tls1_2
# added by Michael Hieb 20-Jan-2017 for lmtp on port (not socket)
lmtp_admins: lmtpuser
MAILSERVER:~ #
Note: before I discovered the problem with forking could be triggered
by switching between single threaded and multi-threaded cyrus server,
I thought problem could be related to resource constraints. This is
why I set all processes with maxchild=-1 maxforkrate=100. I also set
the following in /etc/systemd/system/cyrus.service
MAILSERVER:~ # cat /etc/systemd/system/cyrus.service
[deleted...]
LimitRTPRIO=50000
LimitNOFILE=50000
LimitNPROC=50000
MAILSERVER:~ #
None of these resource settings made any difference and the problem
goes away as soon as cyrus is single threaded - so I do not believe it
is a resource constraint (at least one I understand at this point).
Any help would be much appreciated.
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
