On 6/10/05, Rhesa Rozendaal <perl&nntp@xxxxxxxxx> wrote:
> Hello all,
>
> I need some help debugging an issue with Apache2 becoming completely
> unresponsive. Any help, tips and suggestions most welcome!
>
> I'm running a light-weight Apache2 server configured as a reverse proxy,
> and serving some static content. Traffic is normally moderate at 2
> requests per second. Occasionally though, a sudden spike shows up in the
> traffic (over 200 concurrent tcp connections being made in a very small
> timespan).
>
> Some of those spikes (but not all, oddly enough; this only happens every
> other day or so) cause Apache to become completely autistic. It no
> longer serves requests, and apachectl -k graceful does nothing to remedy
> that. Only a complete restart brings it back from its "zombie" state. It
> doesn't crash (all 150 children are still listed in ps), and it does
> eventually (after a couple of hours) come back alive. I've written a
> little monitoring script that restarts it whenever this happens, so
> downtime is minimal.
>
> As far as I can tell, traffic patterns around the time of these
> blackouts are not particularly different from normal.
>
> I have no idea what is going on at those odd moments. What can I do to
> make sure Apache stays alive? What can I do to debug this?
I don't have much experience in debugging these types of problems, but
here are a few guesses:
- You are sure that this isn't caused by some deliberate or accidental
DoS? Have you examined netstat during these periods to see what is
hitting apache? Does raising MaxClients help (providing you have
enough memory)? If you are able to connect at all to apache, the
output of the server-status page might be helpful.
- A wild guess would be to try changing the AcceptMutex. If the
clients get stock here, it could cause your server to look dead.
- For debugging, you can start by strace'ing the children, but you may
also need to try attaching to them with a debugger to see what they
are really up to.
- mod_log_forensic is good for identifying if there is something
particular about a request that is causing problems:
http://httpd.apache.org/docs-2.0/mod/mod_log_forensic.html
Joshua.
>
> TIA,
> Rhesa Rozendaal
>
>
> Some config information:
> OS: Fedora Core 3
> RAM: 1GB ( always at 80% used, no swap used)
> Load average: 0.03 0.06 0.01
> Server version: Apache/2.0.52
> Server built: Nov 11 2004 10:31:42
>
> What I think is relevant from httpd.conf:
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 10
>
> # prefork MPM
> StartServers 35
> MinSpareServers 30
> MaxSpareServers 50
> ServerLimit 180
> MaxClients 150
>
> LoadModule log_config_module modules/mod_log_config.so
> LoadModule mime_magic_module modules/mod_mime_magic.so
> LoadModule setenvif_module modules/mod_setenvif.so
> LoadModule mime_module modules/mod_mime.so
> LoadModule status_module modules/mod_status.so
> LoadModule negotiation_module modules/mod_negotiation.so
> LoadModule dir_module modules/mod_dir.so
> LoadModule alias_module modules/mod_alias.so
> LoadModule rewrite_module modules/mod_rewrite.so
> LoadModule proxy_module modules/mod_proxy.so
> LoadModule proxy_http_module modules/mod_proxy_http.so
>
> MaxRequestsPerChild 40000
> ProxyRequests Off
> ProxyReceiveBufferSize 131072
> ProxyIOBufferSize 131072
> ProxyVia On
> ProxyPreserveHost On
> <Proxy *>
> Order deny,allow
> Allow from all
> </Proxy>
>
> RewriteEngine On
> RewriteCond /var/www/html/%{REQUEST_FILENAME} -f
> RewriteRule ^/(.*) /var/www/html/$1 [L]
> # if not a static file, pass through to the backend mod_perl machine
> RewriteRule ^/(.*) http://192.168.42.42/$1 [P,L]
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|