Eliie, thank you for that information about imapd usage. I have only
been managing our imap for a few months and am still learning.
We do not have a lot of active users on each server (probably under 20),
but enough that what you describe could be happening.
As for how I am determining they are reaching their usage limit, all I
have are the process start times. When we hit the 100 limit I can see
that the preforked 5 are still running as are many that were started
many hours ago. I could be crazy wrong (probably am ;-) but it seemed to
me that if processes were being killed and recycled I should not see a
lot of old processes sitting around.
And I owe an apology to Konrad...I originally wrote this email a week
ago and then set it aside while I tried setting tcp_timeout=1. Then
when that did not help, I sent the email and forgot to update my
imapd.conf. This is what I have in my imapd.conf.
tcp_keepalive: 1
tcp_keepalive_idle: 900
tcp_keepalive_cnt: 5
tcp_keepalive_intvl: 75
It would be great if anyone who is using tcp_keepalive could share their
settings in case I completely misunderstood how these work.
In the meantime I am going to bump my max processes up to 200 and my
monitoring warning level to 150 see how that works.
Thanks for all the help everyone.
Andy
On 09/11/2016 11:05 PM, ellie timoney via Info-cyrus wrote:
We now have 4 of the 14 servers where the number of imapd processes
slowly grows and usually within 12-14 hours reaches the imapd process
limit (currently set to 100).
How many user accounts are on these servers? Each client connection
constitutes one imapd process on the server.
It's my understanding that Thunderbird (for example) in its default
configuration holds 5 connections open at once, and tries to keep them
open (either with IDLE, if enabled, or NOOP), and will reconnect them
after a while if they disconnect. (I would expect most other IMAP
clients of a similar maturity to behave similarly.) So it would only
take 20 users leaving their mail client open most of the time to hit
your limit of 100 concurrent imapd processes.
So 100 seems like a very small number, unless you have a very small
number of user accounts.
From what we can tell watching the
process list, the old processes are never shut down after reaching their
usage limit (which is the default).
Do you have more information about how you determined that their usage
limit had been reached?
While we had this problem intermittently with 2.4.18 (once or twice a
month on a random server), it is much worse since the update to 2.5.9.
This might simply be 2.5.9 doing a better job of helping clients keep
open connections open?
Cheers,
ellie
On Mon, Sep 12, 2016, at 01:01 PM, Andy Dorman via Info-cyrus wrote:
Hi everyone. We have 14 Debian servers running the current Debian
release of Cyrus imap 2.5.9.
We upgraded from 2.4.18 to 2.5.9 on Aug 28, upgraded the cyrus.conf &
imapd.conf to rename the deprecated options and switch from skiplist to
twoskip for all our dbs. Our cyrus.conf and imapd.con (minus comments)
is at the end of this email.
We now have 4 of the 14 servers where the number of imapd processes
slowly grows and usually within 12-14 hours reaches the imapd process
limit (currently set to 100). From what we can tell watching the
process list, the old processes are never shut down after reaching their
usage limit (which is the default).
The difference between the 4 with the problem and the others have to be
the mail clients. Each server supports a different group of mail
domains and users, so I suspect some of the users on the problem servers
are using one or more mail clients that are not well behaved, but I have
no indication of what they are doing wrong.
While we had this problem intermittently with 2.4.18 (once or twice a
month on a random server), it is much worse since the update to 2.5.9.
We now have to restart imapd on all the problem servers almost twice a
day.
If it matters, we have tried both idled and polling and have not seen
any difference.
So has anyone else seen an issue like this? Does anyone have any
suggestions on how to debug?
We have examined syslog and do not see any strange reports outside of
the normal berkeley DBERROR messages we always see due to bdb compiled
into the debian Cyrus release.
===== cyrus.conf =====
START {
recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r"
idle cmd="idled"
delprune cmd="/usr/sbin/cyrus expire -E 3"
tlsprune cmd="/usr/sbin/cyrus tls_prune"
}
SERVICES {
imap cmd="imapd" listen="*:imap" prefork=5 maxchild=100
imaps cmd="imapd -s" listen="*:imaps" prefork=5 maxchild=100
lmtp cmd="lmtpd" listen="*:lmtp" prefork=5 maxchild=20
sieve cmd="timsieved" listen="*:sieve" prefork=0 maxchild=100
}
EVENTS {
checkpoint cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30
delprune cmd="/usr/sbin/cyrus expire -E 3" period=120
tlsprune cmd="/usr/sbin/cyrus tls_prune" period=60
resquat cmd="/usr/sbin/cyrus squatter -s -r -i *" at=0437
}
===== imapd.conf =====
admins: cyrus
allowallsubscribe: on
allowanonymouslogin: no
allowplaintext: on
altnamespace: on
annotation_db: twoskip
anyoneuseracl: off
autocreate_quota: 512000
configdirectory: /var/lib/cyrus
defaultdomain: ironicdesign.com
defaultpartition: default
duplicate_db: twoskip
expunge_mode: delayed
fulldirhash: on
guid_mode: sha1
hashimapspool: on
idlesocket: /var/run/cyrus/socket/idle
improved_mboxlist_sort: on
lmtp_downcase_rcpt: on
lmtpsocket: /var/run/cyrus/socket/lmtp
mboxname_lockpath: /run/cyrus/lock
notifysocket: /var/run/cyrus/socket/notify
partition-default: /var/spool/cyrus/mail
popminpoll: 1
proc_path: /run/cyrus/proc
quota_db: twoskip
sasl_log_level: 7
sasl_mech_list: PLAIN
sasl_pwcheck_method: saslauthd
sasl_minimum_layer: 0
allowapop: no
sieveusehomedir: false
sievedir: /var/spool/cyrus/sieve
statuscache_db: twoskip
syslog_prefix: cyrus
tls_sessions_db: twoskip
tls_client_ca_file: /etc/ssl/certs/mail.ironicdesign.com.pem
tls_client_ca_dir: /etc/ssl/certs
# The rest of our TLS setup
tls_server_cert: /etc/ssl/certs/mail.ironicdesign.com.pem
tls_server_key: /etc/ssl/certs/mail.ironicdesign.com.pem
tls_session_timeout: 1440
tls_ciphers: TLSv1+HIGH:!aNULL:@STRENGTH
tls_versions: tls1_0 tls1_1 tls1_2
umask: 077
unix_group_enable: off
unixhierarchysep: on
userdeny_db: twoskip
virtdomains: userid
sieve_extensions: fileinto reject vacation imapflags notify envelope
body relational regex subaddress copy
===== end imapd.conf =====
Thanks for any ideas on what to look for or how to debug this.
--
Andy Dorman
Ironic Design, Inc.
AnteSpam.com
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
