Stephan,
Thank you for this very interesting description of both the problem
and the solution. I'm curious which Linux distribution you're
using? We're using Ubuntu Xenial (16.04) and find the following
default settings:
$ grep -i tasks /etc/systemd/system.conf
#DefaultTasksAccounting=no
#DefaultTasksMax=
so this would seem not to be an issue on Ubuntu systems, at least.
Cheers,
-nic
On 12/16/2016 02:54 AM, Stephan Lauffer
via Info-cyrus wrote:
Hello!
While testing 2.5 we struggled over (known) problems like this...:
...ssh can't fork process to run service imaps/ipv4: Resource
temporarily unavailable
I searchd... and I found your FAQ...
https://cyrusimap.org/imap/faqs/o-toomanyprocesses.html
...and this "tcp_keepalive = 1" hint was doing a good job, but not
fixing our problem completely. Less processes as without this
setting but too many at all.
Btw we have a murder with 5 backends and each has about 5k mboxes.
With 2.4 we have about 400 imapd processes on the backends (in
normal state) and about 800 on the proxy. Peaks raises this values
for sure.
But we still wasn't able to get over a limit of about 500 imapds
on our new system which is running 2.5. Yesterday night I found
our problem:
SYSTEMD
(and not cyrus-imapd-2.5, but 2.5 was running with systemd
here...)
I guess there may be more people out there which may be interested
in a solition (we will add fixes to our systemd scripts of our
suse bulds soon, see
https://build.opensuse.org/package/show/home:nixda:devel/cyrus-imapd).
The first problem there is:
Systemd does not care about your /etc/security/limits.conf
The second problem is:
Systemd comes with a new limit called "TasksMax". You will find
this limit f.e. in /etc/systemd/system.conf as
"DefaultTasksMax=512". But you will NOT find this limit in "cat
/proc/`cat /var/run/cyrus.pid`/limits".
The solution is to add a file with a systemd override for this
service. F.e. use the systemctl edit <sevice> command...
(like "systemctl edit cyrus-imapd.service") or place a file in
/etc/systemd/system/cyrus-imapd.service.d/override.conf with f.e.:
[Service]
TasksMax=2048
LimitNOFILE=10000
The LimitNOFILE is just a guess... maybe you may not need this
here.
After setting this override you will see a status information
about your process count and limit like this:
# systemctl status cyrus-imapd.service
● cyrus-imapd.service - The Cyrus IMAP and POP Mail Server
Loaded: loaded (/usr/lib/systemd/system/cyrus-imapd.service;
enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/cyrus-imapd.service.d
└─override.conf
Active: active (running) since Thu 2016-12-15 20:51:17 CET; 12h
ago
Main PID: 12901 (master)
Tasks: 326 (limit: 2048)
[...]
What about adding a short hint to your faq? Yes, it is not a cyrus
problem. And yes, a sysadmin should know... but... I guess systemd
in depth may not be good known as the old ulimit thing and if you
are searching in the web 99% of the hits deal with ulimit.
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Nic Bernstein nic@xxxxxxxxxxx
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
|
begin:vcard
fn:Nic Bernstein
n:Bernstein;Nic
org:Onlight, Inc.
adr:Suite 24;;6525 W Bluemound Road;Milwaukee;WI;53213-4073;USA
email;internet:nic@xxxxxxxxxxx
title:VP Operations
tel;work:414-272-4477 x204
tel;cell:414-807-1734
url:http://www.onlight.com/
version:2.1
end:vcard
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
