RE: Apache 2.x configuration for high load servers

Rob Morin <rob@xxxxxxxxxxxx> · Wed, 25 May 2011 15:00:15 -0400

Thanks for the prompt reply Mark, I will try to add a few things and clear
up some stuff I might have omitted in my first post...

I have server-status enabled as well as Munin for Apache..

I have attached a munin graphs to this email, not sure if its proper or
not... sorry if not... one is accesses and one is processes per day..
To be safe its about 120 accesses per second , each server is basically the
same give or take 5%, I will enable access logging for one domain and see
what happens with respect to the time to serve a request...

Here is a paste of server status for the same server as the attached pngs of
munin.. our busy time will start in the next couple hours....

Current Time: Wednesday, 25-May-2011 13:48:42 EDT
Restart Time: Sunday, 22-May-2011 04:02:05 EDT
Parent Server Generation: 1
Server uptime: 3 days 9 hours 46 minutes 37 seconds
Total accesses: 24212224 - Total Traffic: 7.1 GB
CPU Usage: u1544.11 s494.57 cu0 cs0 - .692% CPU load
82.2 requests/sec - 25.5 kB/second - 316 B/request
46 requests currently being processed, 384 idle workers

_C____._..____._..__C___..__C___.___.____._.._.__._._._..____.__
__C_._..____._.__.____________.___._______________.___._.__.R.__
._C____CC___C_.__CCR.._._..WW_____.._.C_..C..__.._.__.._______._
__._..__._.__.____.C__C._____..__.__C.____C___._______.C______._
.C_._._.__.________C._____.._..__._..__.C_C__.._.___.__.._____C_
_____._._C__.________.._____.__R_.____.._C_.C.__.CCC.______.__._
._C_____....__.___..__.____.__.__.____._____...____._._._.__.__C
._C______..______._._._________._.___..____....CC_.__C_C_.C__._.
R__...C.__._.___..._.___C_______....__._____._____._____..._.._C
__W.__.__.________..___.

So then my current config I have seems good enough then, from what you have
spoken about so far? 

StartServers       150
MinSpareServers    125
MaxSpareServers   250
ServerLimit      600
MaxClients       600
MaxRequestsPerChild   1500

As the load at busy times is about 9, and this is for quad core, so there
might be room for improvement somewhere I guess.. This is why I turned off
logging as we have about 60 domains all writing logfiles via rotatelogs I
fogured removing that extra file open and writing to would help a bit... I
also made a small RAMdisk to store the eAccelerator cache files rather than
use disk... we also use memcache to store sessions and for mysql querries..
We also use Edgecast as our CDN for static content, like images and CSS
files..

We also use New Relic and new PHP performance tuning  tool, its very cool...
that shows that a good 95% of our requests take under 200 ms while the rest
can vary between 600 ms and way up to 10 seconds, the 10 second ones are
where we connect to processors to valaidate information on clients..

I will read over that in the URL your pasted and see what I can do to still
increase performance...

As for the logs just to clarify how I did it, all the virtual host config
files have the CustomLog entries commented out, with this in minf that means
Apache will send access log detaisl to whatever the defautl access log file
is , its normally something like /var/log/hhpd/access_log, so in the
httpd.con file under CustomLog combined , I just have %h, then in the
location for customlog rather than /var/log/httpd/access_log I have
/dev/null so this way it will og minimum info and then not write it to disk
and send it out to the bit bucket.... we do nto need logs as we use Google
Analytics in our code. Is this ok, as if I just remove or comment out the
access_log loacation or have no combined entry apache will nto restart....
if I remove mod_log then I will get no errors logs, and those are more
important than access_logs for us...

Hope this sheds more light on my problem...

Thanks again for all your help...

Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207

-----Original Message-----
From: Mark Montague [mailto:mark@xxxxxxxxxxx] 
Sent: Wednesday, May 25, 2011 12:06 PM
To: users@xxxxxxxxxxxxxxxx
Cc: Rob Morin
Subject: Re:  Apache 2.x configuration for high load servers

  On May 25, 2011 10:27 , Rob Morin <rob@xxxxxxxxxxxx> wrote:
> We have recently had a 30% increase in traffic and will be expecting 
> more. I would like to know the correct way to calculate the proper 
> settings for prefork.c my settings, here is what I have now...

> I am going to add more RAM today to make a total of 24 gigs on each 
> server, and i am going to add an additional web server in the next 
> week or so...
>
> Each day we get about 5 million visits...
>

The settings you use for the prefork MPM are not determined by how many 
visits you get per day, but by how many concurrent requests the server 
in question needs to handle during your busiest periods. Number of 
concurrent requests is in turn determined by how quickly requests come 
in together with how long it takes to serve (process, respond to, 
handle) each request.

For example, let's say that 90% of the requests you serve require 1 
second to serve, but 10% of the requests take 5 seconds to serve. Let's 
also assume that during your busiest period you are getting 100 requests 
per second.
In each second, then, 100 children will be needed to server these 
requests -- 90 of these children will finish (and become available to 
serve another request) after 1 second and the remainder will become 
available again after 5 seconds. This means that after 5 seconds of your 
busiest anticipated load, 140 children will be tied up serving requests, 
and hence you'd want to set ServerLimit comfortably above this value (to 
allow margin for error and future growth), assuming that you have the 
necessary hardware resources (RAM, processor cores) to effectively do 
so. In addition, you may want to set MinSpareServers high enough to 
handle a spike in load equivalent to one second's worth of requests, and 
set MaxSpareServers to free up resources if more than the number of 
children required to server two seconds worth of requests are sitting idle.

Access logs with a custom log format containing %D (time required to 
serve the request in microseconds) are your best tool for figuring out 
the request patterns that your web server needs to be able to handle. 
Also, mod_status is a useful tool for seeing what each child process is 
doing at single point in time (which children are serving requests, 
which children are idle, and so on).

> Apache is currently using 6207 Megs of memory...
>
> There are 245 Apache processes running at this time.
>
> --------------------------------
>
> So the load is high, but i have free RAM left and only running 245 
> Apache processes, out of a possible 600? I only show the first few 
> lines of top ..
>

This is anticipated. Apache HTTP Server's prefork MPM will try to keep 
the number spare processes (children that are idle and waiting for new 
requests) between MinSpareServers and MaxSpareServers at all times.

The rule of thumb I recommend is that you want your system load to be 
less than twice the number of processor cores. Other people's advice may 
differ from this, so I recommend that you ask around.

If your system load is more than twice the number of processor cores, I 
recommend:

- Determine if there is any storage (disk, etc.) I/O bottleneck; if so, 
eliminate it.
- Determine if there is any network I/O bottleneck (network link, load 
balancer, firewall, etc.); if so, eliminate it.
- Reduce the time required to serve each requests:
-- Reduce time required by any web applications run by the web server 
(active content)
-- Reduce time required by the web server itself (web server 
configuration, operating system configuration)

The above recommendations may result in you adding more I/O capacity, 
more RAM, more processor cores, and/or more machines to your web server 
cluster. However, I believe it is generally better to only add more 
resources once you have determined that all existing resources are being 
used as optimally as possible: improve any web applictions or other 
active content on your web site so that they require less resources; 
change your infrastructure architecture (what each web server does and 
how it does it including caching and proxying); and improve the 
configuration or your web server (and other servers, including any 
database servers) and your operating system.

> Or do I get this wrong, does each connection have an httpd process 
> that is run, or are child/subsequent connections, not listed on under 
> a ps ax??
>

If you are using the prefork MPM, a number of child processes get 
created to handle web server requests (connections). The number of these 
will vary dynamically based on how you configure the prefork MPM and 
also on how many requests come in how quickly. All of the processes will 
show up under ps.

> I removed access logging, I disable access logging by commenting out 
> the CustomLog directive in each virtualhost and then in httpd.com I 
> set CustomLog value to /dev/null, this way I can keep error working 
> too. is this ok? Here aer some of the httpd.con entries.
>

Are you sure that you want to not log web server accesses? This removes 
an incredibly important diagnostic tool. Also note that if you use 
CustomLog with a file of /dev/null, then you are having Apache HTTP 
Server actually create each log entry and send it to the kernel, which 
will then discard it instead of writing it to disk; if you really want 
to turn off access logging, then it would be better to completely remove 
any CustomLog or TransferLog directives from your web server configuration.

If you want to improve performance, see the recommendations at 
http://httpd.apache.org/docs/2.2/misc/perf-tuning.html

> Here are loaded modules, I am not sure if I need them all, can I 
> safely turn any off?

You can and should turn off any modules you are not using; this will 
save some memory and may make your web server a little more secure. You 
will need to look at what functionality your web site currently relies 
upon, together with your web server configuration, in order to determine 
which modules you can avoid loading.

--
   Mark Montague
   mark@xxxxxxxxxxx

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx