Re: Apache 2.x configuration for high load servers

[Bostjan Skufca <bostjan@xxxxxx>]

Hi Rob,

couple of thoughts below, please inform me if I make wrong deductions somewhere, tnx.

1. Your static content is served from elsewhere, you are only talking about application servers
2. Your average response time is 200ms, and given 4cores per server and excluding delays you are currently processing 20 requests per second per server tops.
3. If your load is hitting above number of cores your requests are not served as fast as possible
4. Keepalive timeout, what is your setting? Given the presumption that you don't serve static content yourself it means that you only serve master html - you don't need it at all (with given circumstances of course, where each open connection in keepalive state consumes one apache process)
4a. On the second thought maybe you can use keepalive between load balancer and APP servers, to avoid additional delay of connection setup
5. Did you consider compiling apache yourself? Build it statically with all modules compiled in (and as DSO only those modules you need occasionally) and with -O3? The latter calls for careful testing. Same goes for PHP
6. Did you try profiling your application? Use xdebug to make a request trace and start optimizing there, usually it is far more efficient that system-level optimization.
7. Memcache is slow compared to file-based cache, given that you have enough ram to hold all your files in cache. We've performed the tests and unless you don't need distributed cache filesystem cache is better/faster solution.
8. Maybe latencytop is the next optimization tool, or ftrace if you are using linux.
9. IO graphs (IOPS, bytes per seconds, disk time per second, time per IOP), do you have them?
10. Can you provide also load, cpu utilization, network bytes and packets, and memory usage graphs?

b.

On 27 May 2011 14:24, Rob Morin <rob@xxxxxxxxxxxx> wrote:

Thanks for all your help... first off I increased MaxRequestsPerChild from
1500 to 10000 on one webserver, just as a test to make sure all is good....

As for KeepAlive, we were told we cannot use it due to we are also using
keepalived on our haproxy load balancer, is this true?? As we have a bunch
of SSL certs

I already had ServerToken set to prod :)

I am not sure about UseCanonicalName On, as from what I read it might break
our rewrite rules we use them alott

So I commneted out mod_dir and Apache would not start I got this error, so I
put it back..

Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf:
Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module
not included in the server configuration

I will disable atime later on this morning, I will mod fstab and reboot one
server at a time... the load balancer will autoamtiocally remoe that server
form the cluster when its down so ti will not affect any clients.. :)

I will read up more on mod_file_cache

Thanks again...

Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207




-----Original Message-----
From: Geoff Millikan [mailto:gmillikan@xxxxxxxxxxxxx]
Sent: Friday, May 27, 2011 3:35 AM
To: users@xxxxxxxxxxxxxxxx
Subject: RE: Apache 2.x configuration for high load servers

#Increase this number.  The zombies you're seeing
#is every time an Apache child process dies.
#The child will die @1500.  We run at 20000 and are having no problems.
MaxRequestsPerChild   1500

#We don't run as hot as you (we average 3 hits/sec and max at like 15
#in a 24 hour period) but here's our settings:
StartServers    70
MinSpareServers 70
ServerLimit             364
MaxClients              364
MaxRequestsPerChild  20000

#Shrink the size of the response header:
ServerTokens Prod

#Defiantly want keepalive on, it will help page load time for customers.
#Everything else looks good.
Keepalive is on

#We have this turned on but we use server side includes SSI and we cache.
UseCanonicalName On

You have way too many modules enabled.  You need user_dir module?  Every
module loaded takes up RAM.  You should be able to get the
"RES" RAM use per process down to under 20 MB unless you're doing something
funky.

If you're finding that 80% of the time you're running 100 processes, then
start that many and keep that many going.  We found that
the process that starts up new children takes too long and it was causing
pages to hang while Apache started up new children.   So
just start as many as you need, even if a lot of them are just idle.

Couple of other tips:

1. Turn off the "access time" atime on your file system.  This will speed
disk access a lot.

2. Using mod_file_cache on a big site is hard to get everyone on board for.
It's complicated and it's easy to cache the wrong
stuff.  However, with a bit of pain, there is a lot of gain to be had there
in terms of reduced work load on the server and thus
faster page load times.  Suggest you bite the bullet and do it.  BTW, the
CentOS rpm of Apache doesn't work with mod_file_Cahce.
Lots of bugs.  So have to compile from source on this.  We're on CentOS 5
too.

Lots of people will suggest the Worker MPM due to it's lower RAM usage
however we haven't done it as the PHP web site speaks so
strongly against it.

Best,

Geoff Millikan @ http://www.t1shopper.com/


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
  "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
  "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx