Re: Runaway Apache Process

[Dan Bunyard <danodemano@xxxxxxxxx>]

I have never done a backtrace, can you please point me in the right direction for that?

I didn't check CPU usage at the time, only load average which was around 100 (normally it's between 0.02 and 0.5 over 1 minute).

I was able to log in but it was VERY slow.  As I watched the load average it was continuing to climb just before I killed Apache.  It did not terminate gracefully either, the error_log showed this:
[Mon Jan 25 12:50:49 2010] [warn] child process 23437 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:49 2010] [warn] child process 23441 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:49 2010] [warn] child process 23445 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:49 2010] [warn] child process 23451 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:49 2010] [warn] child process 23453 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:49 2010] [warn] child process 28350 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:49 2010] [warn] child process 28355 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:49 2010] [warn] child process 26939 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:51 2010] [warn] child process 23437 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:51 2010] [warn] child process 23441 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:51 2010] [warn] child process 23445 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:51 2010] [warn] child process 23451 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:51 2010] [warn] child process 23453 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:51 2010] [warn] child process 28350 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:51 2010] [warn] child process 28355 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:51 2010] [warn] child process 26939 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:53 2010] [warn] child process 23437 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:53 2010] [warn] child process 23441 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:53 2010] [warn] child process 23445 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:53 2010] [warn] child process 23451 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:53 2010] [warn] child process 23453 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:53 2010] [warn] child process 28350 still did not exit, sending a SIGTERM

[Mon Jan 25 12:50:53 2010] [warn] child process 28355 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:53 2010] [warn] child process 26939 still did not exit, sending a SIGTERM
[Mon Jan 25 12:50:55 2010] [error] child process 23437 still did not exit, sending a SIGKILL

[Mon Jan 25 12:50:55 2010] [error] child process 23441 still did not exit, sending a SIGKILL
[Mon Jan 25 12:50:55 2010] [error] child process 23445 still did not exit, sending a SIGKILL
[Mon Jan 25 12:50:55 2010] [error] child process 23451 still did not exit, sending a SIGKILL

[Mon Jan 25 12:50:55 2010] [error] child process 23453 still did not exit, sending a SIGKILL
[Mon Jan 25 12:50:55 2010] [error] child process 28350 still did not exit, sending a SIGKILL
[Mon Jan 25 12:50:55 2010] [error] child process 28355 still did not exit, sending a SIGKILL

[Mon Jan 25 12:50:55 2010] [error] child process 26939 still did not exit, sending a SIGKILL
[Mon Jan 25 12:50:56 2010] [notice] caught SIGTERM, shutting down

Is there a way to turn on more logging (debug logs) or a better way to trace what it doing at that time?

Thanks,  
--
Dan

http://www.moonlightrpg.com
http://www.linkedin.com/in/danbunyard
http://www.danodemano.com

http://www.dansrandomness.com
http://www.danandshelley.com

This is not a problem that requires infinite wisdom, Benj. This is a problem that requires enough neural organization to qualify as a vertebrate, apparently a stretch for some folks these days.

~Cecil Adams.

On Wed, Jan 27, 2010 at 08:18, Jeff Trawick <trawick@xxxxxxxxx> wrote:

On Tue, Jan 26, 2010 at 8:28 PM, Dan Bunyard <danodemano@xxxxxxxxx> wrote:
> This has happened twice now and it's a little bit concerning to me. I have a
> Fedora 12 server with 5GB of RAM that I use to host a few small web sites of
> mine. As I mentioned, this happened once before. I tried to load one of my
> web sites today and it took FOREVER (as in the 10s of minutes) to load. I
> SSHed into the box and found the load average around 100 (dual core
> machine). Since this was the second time it had happened, I knew that it was
> Apache causing it. So I restarted the Apache service and everything returned
> to normal. A look in the error_log showed this error:
>
> server reached MaxClients setting, consider raising the MaxClients setting
>
> I suspect that this is the reason that Apache was eating up all my system
> resources but I don't have any idea how to fix it.

This means that you have 100 active client connections, and that's the
limit of your configuration (MaxClients=100).

I didn't catch whether or not you had high CPU utilization.

I didn't catch whether or not you had a high number of requests being
processed during this time.

High CPU utilization, relatively low number of requests:

I'd guess that some application code running inside Apache encounters
an unexpected situation that results in loops or other extremely high
CPU that prevents the request from being completed within a reasonable
period of time (or ever).  The fact that you could log in after a
while suggests that some of this faulty request processing does
eventually finish.

High CPU utilization, relatively high number of requests:

Your server is just being overwhelmed -- application request
processing requires noticable CPU, and the box can't handle large
numbers of concurrent requests.  Likely some application-level
optimization will help.


If you pick an httpd child process and get backtraces of it at
intervals with gdb to see where it is spending its time, that might
provide valuable clues.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
  "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx