Re: Apache intermittent crash - HELP!!!

[gary lim <garylim.in@xxxxxxxxx>]

Hi,

1. Unfortunately, our website depend heavily on PHP to operate, so it not possible for us to disable it. We also highly suspect that the crash is related to one of our php module because we are running a lot of Joomla component.

2. unfortunately, we are unable to track which module is causing this issue. Any idea how we can check which script is causing the problem because this has been going on for weeks and we are still unable to determine the source.

3. Kindly advise how we can perform the migration of our existing codebase to FastCGI/SCGI without breaking the current website?


rgds

On Thu, Dec 17, 2009 at 12:28 AM, Tom Evans <tevans.uk@googlemail.com> wrote:

On Wed, Dec 16, 2009 at 4:20 PM, gary lim <garylim.in@gmail.com> wrote:
>
> Dear all,
>
> 1. I'm using Linux 2.6.16.29 and running on i686 Intel(R) Pentium(R) 4 CPU 3.00GHz GNU/Linux
> 2. The below info are my machine config and log output
> 3. Kindly advise how to resolve intermittent apache crash even after performance tuning
>
> httpd.conf output
> # prefork MPM [DEFAULT IF USE=-threads]
> # StartServers: number of server processes to start
> # MinSpareServers: minimum number of server processes which are kept spare
> # MaxSpareServers: maximum number of server processes which are kept spare
> # MaxClients: maximum number of server processes allowed to start
> # MaxRequestsPerChild: maximum number of requests a server process serves
> <IfModule prefork.c>
>     StartServers         2
>     MinSpareServers      4
>     MaxSpareServers     6
>     MaxClients         100
>     MaxRequestsPerChild  500
> </IfModule>
>
> # worker MPM [DEFAULT IF USE=threads]
> # StartServers: initial number of server processes to start
> # MaxClients: maximum number of simultaneous client connections
> # MinSpareThreads: minimum number of worker threads which are kept spare
> # MaxSpareThreads: maximum number of worker threads which are kept spare
> # ThreadsPerChild: constant number of worker threads in each server process
> # MaxRequestsPerChild: maximum number of requests a server process serves
> <IfModule worker.c>
>     StartServers         2
>     MaxClients         100
>     MinSpareThreads     25
>     MaxSpareThreads     75
>     ThreadsPerChild     25
>     MaxRequestsPerChild  500
> </IfModule>
>
>
>
> top output
> top - 00:09:00 up 157 days, 23:12,  2 users,  load average: 0.67, 0.85, 0.80
> Tasks:  47 total,   1 running,  46 sleeping,   0 stopped,   0 zombie
> Cpu(s):  8.3% us,  1.4% sy,  0.0% ni, 88.0% id,  2.2% wa,  0.0% hi,  0.0% si
> Mem:    917652k total,   858880k used,    58772k free,   142820k buffers
> Swap:  1048568k total,    37844k used,  1010724k free,   434476k cached
>
> apache2 error log
> *** glibc detected *** /usr/sbin/apache2: double free or corruption (fasttop): 0x08b4faf0 ***
> [Tue Dec 15 13:04:22 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
> *** glibc detected *** /usr/sbin/apache2: double free or corruption (fasttop): 0x086a1170 ***
> *** glibc detected *** /usr/sbin/apache2: double free or corruption (fasttop): 0x08b48c08 ***
> [Wed Dec 16 23:53:07 2009] [warn] child process 1325 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:07 2009] [warn] child process 19910 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:07 2009] [warn] child process 10569 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:07 2009] [warn] child process 16964 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:07 2009] [warn] child process 19718 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:09 2009] [warn] child process 1325 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:09 2009] [warn] child process 19910 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:09 2009] [warn] child process 10569 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:09 2009] [warn] child process 16964 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:09 2009] [warn] child process 19718 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:11 2009] [warn] child process 1325 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:11 2009] [warn] child process 19910 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:11 2009] [warn] child process 10569 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:11 2009] [warn] child process 16964 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:11 2009] [warn] child process 19718 still did not exit, sending a SIGTERM
> [Wed Dec 16 23:53:13 2009] [error] child process 1325 still did not exit, sending a SIGKILL
> [Wed Dec 16 23:53:13 2009] [error] child process 19910 still did not exit, sending a SIGKILL
> [Wed Dec 16 23:53:13 2009] [error] child process 10569 still did not exit, sending a SIGKILL
> [Wed Dec 16 23:53:13 2009] [error] child process 16964 still did not exit, sending a SIGKILL
>
>
> Joomla / PHP / Apache
> System Information
> PHP built On: Linux trillian 2.6.16.29-ab1-xenU #1 SMP Sat Sep 23 19:20:15 SGT 2006 i686
> Database Version: 5.0.24a-log
> PHP Version: 5.1.6-pl4-gentoo
> Web Server: Apache
> WebServer to PHP interface: apache2handler
> Joomla! Version: Joomla! 1.0.15 Stable [ Daytime ] 22 February 2008 23:00 UTC
> User Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.15) Gecko/2009101601 Firefox/3.0.15 (.NET CLR 3.5.30729)
>
> Mysql
> mysql  Ver 14.12 Distrib 5.0.24a, for pc-linux-gnu (i686) using readline 5.1
>

Does it work if you disable PHP? Does it still exhibit problems if you
run PHP out of process, eg using fastcgi or scgi (can PHP talk SCGI?).
For my money, the double free()s are coming from PHP or a PHP module.

Cheers

Tom

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
  "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx