debugging strategies for httpd that kills the box

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello List,

I have a FreeBSD 5.5 (amd64) server with the following:

Apache/2.0.61 (FreeBSD)
mod_ssl/2.0.53
OpenSSL/0.9.7e-p1
PHP/4.4.4 with Suhosin-Patch
mod_apreq2-20051231/2.6.0
mod_perl/2.0.3
Perl/v5.8.6

Background:  The server was recently updated to FreeBSD 5.5 from 5.3,
which had run for a couple of years without interruption.  I'm not
especially convinced the issue I'm having is related to this upgrade,
but it's the most significant change in some time, so it's worth
mentioning.  Apache is configured with KeepAlive on, as this box
provides a number of other services, but I noticed last week that the
majority of the requests (specifically those being handled by
HTML::Mason via mod_perl) were not sending a Content-Length header,
and thus the KeepAlive settings were largely useless.  I added this
header to the mod_perl output, and began tuning apache to maximize
performance.  I got, for us, quite acceptable results using the
following:

<IfModule prefork.c>
StartServers           5
MinSpareServers        5
MaxSpareServers       10
MaxClients           200
MaxRequestsPerChild    0
</IfModule>

The problem: After a few days the server became unstable.  Twice the
server was out of swap space and had to be rebooted.  While attempting
to solve this problem, I came to understand that the webserver seemed
responsible; for as-yet unknown reason, the box will suddenly lock up,
and if I do not kill httpd within 5 or 10 seconds the server will
become completely unresponsive and need to be rebooted.  If left
alone, swap space will be consumed and the kernel will begin killing
off httpd processes, but not fast enough.  By altering the apache
config to the following, I am able to avoid the hang:

<IfModule prefork.c>
StartServers           5
MinSpareServers        5
MaxSpareServers        5
MaxClients            40
MaxRequestsPerChild    0
</IfModule>

But naturally this provides less than optimal performance.  Other
things I tried, without impact:

-- setting KeepAlive off
-- disabling all vhosts
-- disabling mod_perl2
-- disabling mod_php
-- running httpd -X
-- tweaking/disabling resource limits via Apache2::Resource
-- recompiling httpd
-- recompiling mod_perl2
-- recompiling & reinstalling the freebsd userland

The server has 1gb of ram, 4gb of swap (I recently doubled it in an
attempt to give me more time to diagnose the problem when it occurs --
didn't help, as the lock-up occurs long before swap is consumed), and
in the moments before the hang, shows no particular level of activity
-- swap usage is around 100mb, load is very low (1.5 - 2.0 on a
dual-proc system).  There doesn't seem to be any correlation to
specific requests or request types, whether static, cgi, mod_perl or
php.  There are no errors in the server logs prior to the lock up (
although once, and only once, i saw "httpd in free(): error: page is
already free" ).

I'm hoping someone here can make some suggestions on how to
troubleshoot the problem further, because I'm kind of at a loss.

I have not (yet) posted this to the mod_perl list or freebsd-users,
though I would be happy to do so if it's thought the problem might lie
within those realms.

Thanks!
Greg

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx


[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux