Re: Help troubleshooting performance issue, after "1000 total children" Apache no longer responds to HTTP requests. Not MaxClients issue? (Apache Users)

On Mon, Apr 30, 2012 at 9:13 AM, Alexandr Normuradov <normalex@xxxxxxxxx> wrote:

cat /proc/$(pidof -s httpd)/limitsTo troubleshoot that you should have
at least two additional outputs from

netstat -pant, with connections states
and
service httpd fullstatus, listing current state of all the apache procs/threads.

What applications your Apache is serving?
PHP? is it mod_php, mod_python, mod_perl?

What the vhost access log file for the most accessed vhost is showing?
Any pattern of slow, connections consuming attack?
If it is, and all tasks are in the Keep Alive wait then disable Keep
Alive and lower the general timeout to just 7 seconds.

The error "connect to listener on [::]:80" error is quite unusual.

ETIMEDOUT
Timeout while attempting connection. The server may be too busy to
accept new connections. Note that for IP sockets the timeout may be
very long when syncookies are enabled on the server.

cat /proc/sys/fs/file-nr

cat /proc/$(pidof -s httpd)/limits

Sincerely,
Alexandr Normalex

Hi Alexandr, thanks for taking a look at this with me.

The traffic pattern for this website is at certain times of the day it receives huge spikes of traffic in very short periods of time, trying to tune Apache to accommodate it the best we can.

cat /proc/$(pidof -s httpd)/limits

Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes

Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes

Max resident set unlimited unlimited bytes
Max processes 55296 55296 processes
Max open files 1024 1024 files

Max locked memory 32768 32768 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks

Max pending signals 55296 55296 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0

Max realtime priority 0 0

cat /proc/sys/fs/file-nr
1530 0 560543

Looking at Max open files I see what is likely the problem :)

Max open files 1024

I swear I modified this to 4096! I've changed the limit to 4096 now, I'll double check it tomorrow. Hopefully this will be the obvious fix!

I will check service httpd fullstatus and netstat -pant tomorrow morning when this happens again, it happens the same time every day - it is not an attack, the customers application receives massive amounts of connections at certain times of the day.

I've been working with Apache for 15 years and I've never seen "connect to listener on [::]:80" error message before, I hope it's related to reaching Max open files.

Thanks again for your help.

--
PJ

cat /proc/$(pidof -s httpd)/limits

Limit Soft Limit Hard Limit Units

Max cpu time unlimited unlimited seconds

Max file size unlimited unlimited bytes

Max data size unlimited unlimited bytes

Max stack size 10485760 unlimited bytes

Max core file size 0 unlimited bytes

Max resident set unlimited unlimited bytes

Max processes 55296 55296 processes

Max open files 1024 1024 files

Max locked memory 32768 32768 bytes

Max address space unlimited unlimited bytes

Max file locks unlimited unlimited locks

Max pending signals 55296 55296 signals

Max msgqueue size 819200 819200 bytes

Max nice priority 0 0

Max realtime priority 0 0