Re: Apache 2.2 connections full (Apache Users)

On 06/07/2011 10:59 PM, Rob Morin wrote:

Hello all was not sure how to word my subject line…

I posted a few weeks back on how could I increase performance of apache… I received quite a few replies and they all helped a bit…

We even added an additional server to our server cluster to total 7 web servers now, all load balanced by HAProxy

We currently receive about 14 million impressions a day to about 60 different domains/websites….

Our hosting provider uses HAProxy, rather than an appliance, I assume to reduce costs. However I feel like these HAProxy machines are not reliable at all….Well not under the supervision of our hosting company , anyways…J

On to my questions….

During peek hours, like a few mins ago, all of our web servers became high in load, like close to 100! We still had plenty of RAM left and for sure there was no swapping….

So what was the cause of the load ?

Examine dstat output while the load spikes and see if there is a high percentage spent in IOWait.

Apache does ultimately serve content from disk; is your disk subsystem up to the task ?

Please see my config below as quick snapshot of the httpd.conf file on our Centos Servers.

The weird thing is when I would go see the server-status I would see all the slots filed with either W or C and nothing else…. Then after a what appears a random amount of time it would all clear up , and then come back again. Almost like HAProxy would send a bunch of requests to a server, then it would get full and then(haproxy) say, oh that server is filled I will not send traffic there, for now, then it would come back and see hey this server is empty, lets send a bunch of traffic there…

So it may just as well be the frontend that causes these spikes; you want to establish this with certainty.

Do all apache backend boxes spike at the same time ?

You really need to do a more thorough investigation.

Consider graphing the performance of all boxes, and comparing values during different times of the day.

An rrdtool grapher like cacti or mrtg will help with this, at marginal extra load.

If you consider that too heavy, munin has useful data to correlate CPU with disk I/O and apache stats.

We also use memcache to cache sessions and mysql queries, along with eAcclerator… I assume that those are functioning properly

That's a pretty large assumption. Don't.

Does this make sense, am I on drugs? J

You tell us.

<IfModule prefork.c>
<IfModule worker.c>

This tells us nothing. WHICH ONE are you actually using ?
If you're using prefork (which is the stupid choice for high load machines), starting up a hundred new threads at peak times will take MINUTES.

It looks as if you haven't tuned any of the really vital server variables for this supposed high load you're trying to handle.

<Directory />

Options FollowSymLinks

AllowOverride None

</Directory>

You really want to deny access here.

<Directory "/home/www">

Options -Indexes FollowSymLinks

This does nothing.

    AllowOverride All

    Order allow,deny

    Allow from all

</Directory>

<IfModule mod_userdir.c>

    UserDir disable

</IfModule>

DirectoryIndex index.php index.html

AccessFileName .htaccess

<Files ~ "^\.ht">

    Order allow,deny

    Deny from all

</Files>

TypesConfig /etc/mime.types

DefaultType text/plain

<IfModule mod_mime_magic.c>

    MIMEMagicFile conf/magic

</IfModule>

HostnameLookups Off

ErrorLog logs/error_log

LogLevel notice

LogFormat "%h" combined

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common

LogFormat "%{Referer}i -> %U" referer

LogFormat "%{User-agent}i" agent

CustomLog "/dev/null" combined

That's asenine.

GeoIPEnable On
GeoIPDBFile "/usr/local/share/GeoIP/GeoIPCity.dat"

GeoIPScanProxyHeaders On

Is this in any way a heavy/expensive database ?
What is being logged by this ?
What is it used for ?
Depending on usage, it may be as bad as performing DNS lookups.

-- 
J.