On 18/05/2013 8:14 a.m., Stuart Henderson wrote:
On 2013-05-17, Loïc BLOT <loic.blot@xxxxxxxxxxxxxxxxxx> wrote:
I have found the problem. In fact it's the problem mentionned on my last
mail, is right. Squid FD limit was reached, but squid doesn't mentionned
every time the freeze appear that it's a FD limit problem, then the
debug was so difficult.
Also, i think you must add a non portable check to check if squid FD
limit isn't greater than system ulimit (ulimit -n under OpenBSD). Squid
doesn't mention that and when this appear, it's also a freeze (squid FD
saturation) and no warning is mentionned in the logs. This check would
be at the squid start/reload.
What do you think about it ?
Checking (e.g. via getrlimit) to see if the limit is too small is the
wrong approach. System calls needing FDs can fail if *either* the current
process hits its limits, *or* if the system file table is full.
Various software in OpenBSD has been audited for problems in this
area fairly recently, here are a few of the commits which resulted
showing examples of various approaches we used,
nginx: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=c2d55a68002aae78bcd8ba0198f0a2d3f92cd777
tmux: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=a5b691fe6fa8b648e7d7465167a96f667e40295c
unbound: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=b9781e8c60007fbc60cf8a5509154b0283874682
openbgp: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=bee609c47fa933f06b924435e7f41c661a8a3b40
identd: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=9ca56282c96c6482f23cbd459d2ab9d0cc55a68e
cheers,
Stuart
All of which are checking for and handling ENFILE / EMFILE errors. If
you audit Squid you will find that this has been present since forever.
Squid contains getrlimit() checks at startup/reconfigure to ensure that
the config files and environment are suitably aligned and cut the limit
down to whatever the minimum of those are. ~100 FD are "reserved" below
the actual limit to allow for extra FD which may be necessarily opened
while servicing the N-100'th connection.
When Squid encounters ENFILE/EMFILE at run-time it cuts the limit down
to _below_ the newely detected maximum and *pauses* new inbound
connections until enough FD have been closed to start accepting new
clients. You can see this logged as "Reserved FD adjusted from X to Y"
at the point where FD limits are changed.
Amos
