> So, given that all of these report as idle, should they not be shutting > down at some point? "Idle" has special meaning in IMAP. The last column in the proc/* files is the IMAP command currently being executed by the connected client; "Idle" here means the client is executing the IDLE command (RFC 2177 if you're curious). It does not mean that the connection is unused and could be discarded. Well-behaved modern clients will spend most of their time in IDLE. Another interesting thing you can see from the proc/* files is how long ago the client's current IMAP command was invoked, by looking at the modification time of the proc file (e.g. with ls -l). You'll probably find most of your "Idle" commands were invoked sometime within the last half-hour or so, even though the connection itself may have been up for many hours. If you see files in the Idle state that are more than half an hour old, then that might indicate a client that has entered the Idle state but then lost its connection (which is where the tcp_keepalive settings become useful). > I have the settings below in imapd.conf. My understanding of them is > after a connection has been idle for 15 min, try 5 tcp probes at 75 sec > intervals before marking the connection as broken. This is correct (except note that "idle" here takes the general meaning, not the IMAP meaning). But if the connection is still there, then it's not broken, and tcp_keepalive won't mark it as broken. The fact that the connections aren't being disconnected by your tcp_keepalive settings indicates that the clients are still alive, idling politely until they have something to do (or until the server notifies them of activity). > Also, I see this note in cyrus.conf about idled... So, the client enters the Idle state when it has nothing useful to do. When it wants to execute a new command (maybe because the user has clicked something in the interface), it ends the idle state, sends the new command and deals with whatever response arrives, etc etc. When it has nothing interesting to do anymore (maybe the user has minimized or alt-tabbed away) it re-enters the Idle state. The client can terminate its Idle state and immediately re-enter Idle if it wishes, and will typically do so at least every 29 minutes, so that client and server both know each other are still there. During the Idle state, the server sends the client updates if its mailboxes change (e.g. due to incoming mail, or due to changes made by a different client connected to the same mailbox). This allows the client to get notifications of changes without having to continually ask the server "has anything changed yet" -- you can see how this is good behaviour. On the Cyrus side, without "idled", the Idle state is handled by imapd periodically checking for activity on the relevant mailboxes. How often it does this is determined by the imapidlepoll setting (default: 60 seconds). (This is also the fallback behaviour when there is something wrong with idled.) You can see how a client in this case might not hear about activity until up to 60 seconds after it occurs. Not really problematic, but it can feel slow if you (the user) are expecting an email to appear any second now (maybe you're concurrently on the phone with the sender...). The idled process, if it's running, allows imapd to notify the client instantly when activity occurs, instead of having to wait-check-wait-check-wait-check. It feels a bit faster for the user -- when they expect an email to arrive, boom there it is. It's nice, but not crucial. The "imapidlepoll" setting will completely disable the IDLE command if it is set to 0 (so don't do that). The "idlesocket" setting tells both idled and imapd where to create/look for the socket used for their communication. It doesn't need to match anywhere else, it just needs to be somewhere sensible that the cyrus user has appropriate permissions. Whatever Debian gives you is probably fine. The idled process just needs to be running to be in use -- imapd will use it if it's running and working, and not use it if it's not. To start using it (if you're not already), just enable it in cyrus.conf and restart. It won't make any difference to the number of client connections you see. > # this is only necessary if idlemethod is set to "idled" in imapd.conf > idle cmd="idled" > > But I can't find anything about idlemethod in any of the cyrus docs or > man pages. ... I have no idea where this "idlemethod" business has come from. No such setting exists in Cyrus, and our example cyrus.conf files all have a different comment here: > # this is only necessary if using idled for IMAP IDLE > idled cmd="idled" I can't find any record of "idlemethod" in commit history either, it's not like it's leftover from something that's since changed. Maybe it's from a Debian-specific patch? Anyway, it sounds like your setup is fine, and things are behaving as expected. :) > I know this user so I will ask her what email client she is using and > how often it is set to check email. How often it is set to check email won't make much difference in this case. That is more for servers that don't support Idle, so the client has to keep asking "have you got anything new" every so often. The fact these connections are Idle means it isn't doing that -- it just tells the server it's Idle, and then waits for the server to tell it about any new activity. But understanding how the service is actually being used will do a lot to calibrate your expectations of how the server should be behaving (such as, how many concurrent connections/imapd processes you expect to see) :) Cheers, ellie On Thu, Sep 15, 2016, at 08:07 AM, Andy Dorman via Info-cyrus wrote: > On 09/13/2016 02:15 AM, Bron Gondwana via Info-cyrus wrote: > > On Tue, 13 Sep 2016, at 10:44, ellie timoney via Info-cyrus wrote: > >> Note that it talks about the process being used for new connections. > >> Each imapd process serves one connection at a time (so if you have 50 > >> client connections, you will have 50 imapd processes to serve them, plus > >> whatever you have preforked ready to serve new connections). When one > >> of these connections disconnects, the imapd process that was serving > >> that connection goes back into the pool, ready to serve a new incoming > >> connection -- unless it has already served $uses connections, in which > >> case it exits, and master spawns a new one instead if necessary. > >> > >> So you can see how, if you have mostly long-lived connections, each > >> imapd's count of how many connections it has served would grow very very > >> slowly -- many may not ever reach their limit, because you end up > >> needing to reboot the server (OS security updates, data centre works, > >> etc) before that occurs. > > > > If you look in /var/lib/imap/proc/* you should see one file for each process id, > > which tells you who is connected to that process, and which folder they have > > selected (if any). This is quite useful for tracking down if there's a single user > > causing issues. > > > > Bron. > > > > > > Bron, that was incredibly useful (FWIW, the proc list in Debian is at > /run/cyrus/proc/) > > I took a look and our worst case server appears to have two users > (actually one user with two separate addresses) that are causing the > trouble. I restarted cyrus-imapd about 10 hrs ago and we are up to 28 > imapd processes with start times all through the 10-hr block and they > all appear to be owned by one of two addresses as you can see here: > > [snip addresses] > > So, given that all of these report as idle, should they not be shutting > down at some point? > > I have the settings below in imapd.conf. My understanding of them is > after a connection has been idle for 15 min, try 5 tcp probes at 75 sec > intervals before marking the connection as broken. > > tcp_keepalive: 1 > tcp_keepalive_idle: 900 > tcp_keepalive_cnt: 5 > tcp_keepalive_intvl: 75 > > So is it possible that this client is responding to the tcp keepalive > packets so the idle connection is never shut down, but the client keeps > opening new connections when it wants to check email (which appears to > happen about every 10 - 15 minutes)? > > I know this user so I will ask her what email client she is using and > how often it is set to check email. > > Also, I see this note in cyrus.conf about idled... > > # this is only necessary if idlemethod is set to "idled" in imapd.conf > idle cmd="idled" > > But I can't find anything about idlemethod in any of the cyrus docs or > man pages. All I can find to set in imapd.conf for idle is the socket: > > idlesocket: /var/run/cyrus/socket/idle > > Hmmmmm...wait a minute...in looking for the Debian proc directory I came > across another idle socket created when I restarted cyrus-imapd. > > srwxrwxrwx 1 root root 0 Sep 14 06:10 /run/cyrus/socket/idle > > However, our imapd.conf specified a different idle socket which was also > created when cyrus-imapd was restarted. > > srwxrwxrwx 1 root root 0 Sep 14 06:10 /var/run/cyrus/socket/idle > > So I am thinking I should modify my imapd.conf idlesocket to > "/run/cyrus/socket/idle" which cyrus seems to be setting up anyway. Or > am I completely missing an obvious point here? > > Sorry for being so clueless and thanks for all the help. > > -- > Andy Dorman > Ironic Design, Inc. > AnteSpam.com > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
