cat /proc/$(pidof -s httpd)/limitsTo troubleshoot that you should have
at least two additional outputs from
netstat -pant, with connections states
and
service httpd fullstatus, listing current state of all the apache procs/threads.
What applications your Apache is serving?
PHP? is it mod_php, mod_python, mod_perl?
What the vhost access log file for the most accessed vhost is showing?
Any pattern of slow, connections consuming attack?
If it is, and all tasks are in the Keep Alive wait then disable Keep
Alive and lower the general timeout to just 7 seconds.
The error "connect to listener on [::]:80" error is quite unusual.
ETIMEDOUT
Timeout while attempting connection. The server may be too busy to
accept new connections. Note that for IP sockets the timeout may be
very long when syncookies are enabled on the server.
cat /proc/sys/fs/file-nr
cat /proc/$(pidof -s httpd)/limits
Sincerely,
Alexandr Normalex
On 30 April 2012 10:20, P J <pauljflists@xxxxxxxxx> wrote:
> Greetings all,
>
> Hoping someone can point me in the right direction as I've spent the last
> week trying to figure out where the "issue" is but haven't been able to.
>
> Running Apache 2.2.3 on CentOS 5.8.
>
> At a few points during the day when traffic is heavy we are having an issue
> where Apache no longer responds to any HTTP requests.
>
> It sounded like a standard MaxClients being reached issue, but
> that doesn't seem to be the case.
>
> Also,, logging into the machine during this time the load average is under
> 1, and there is still plenty of RAM available.
>
> Reviewing /var/log/httpd/error_log I've noticed the following pattern:
>
> --snip--
> [Mon Apr 30 07:00:34 2012] [info] server seems busy, (you may need to
> increase StartServers, or Min/MaxSpareServers), spawning 32 children, there
> are 0 idle, and 905 total children
> [Mon Apr 30 07:00:35 2012] [info] server seems busy, (you may need to
> increase StartServers, or Min/MaxSpareServers), spawning 32 children, there
> are 0 idle, and 937 total children
> [Mon Apr 30 07:00:36 2012] [info] server seems busy, (you may need to
> increase StartServers, or Min/MaxSpareServers), spawning 32 children, there
> are 0 idle, and 969 total children
> [Mon Apr 30 07:00:37 2012] [info] server seems busy, (you may need to
> increase StartServers, or Min/MaxSpareServers), spawning 32 children, there
> are 35 idle, and 1001 total children
> [Mon Apr 30 07:00:42 2012] [debug] mpm_common.c(663): (70007)The timeout
> specified has expired: connect to listener on [::]:80
> [Mon Apr 30 07:00:49 2012] [debug] mpm_common.c(663): (70007)The timeout
> specified has expired: connect to listener on [::]:80
> [Mon Apr 30 07:00:56 2012] [debug] mpm_common.c(663): (70007)The timeout
> specified has expired: connect to listener on [::]:80
> [Mon Apr 30 07:01:03 2012] [debug] mpm_common.c(663): (70007)The timeout
> specified has expired: connect to listener on [::]:80
>
> A few times a day, right after "1000 total children" Apache stops responding
> and has to be restarted in order to work again.
>
> I've reviewed the error_log from a few weeks back and it's the same pattern,
> the server hits 1000 total children and then immediately spits out the
> "[debug] mpm_common.c(663): (70007)The timeout specified has expired:
> connect to listener on [::]:80" error message and stops responding.
>
> Yet the load on the server is quite low...
>
> Here is the relevant section from the config:
>
> Timeout 45
> KeepAlive On
> MaxKeepAliveRequests 10000
> KeepAliveTimeout 3
>
> <IfModule prefork.c>
> StartServers 80
> MinSpareServers 50
> MaxSpareServers 120
> ServerLimit 3500
> MaxClients 3500
> MaxRequestsPerChild 2000
> </IfModule>
>
>
> Anyone know why the magic number of children Apache is able to reach is 1000
> before it stops processing more requests?
> Or how to make sense of the "(70007)The timeout specified has expired:
> connect to listener on [::]:80" message?
>
> What "timeout specified" is it referring to? The 45 seconds?
>
> Any help would be greatly appreciated.
>
> Thanks in advance.
> PJ
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|