Re: Apache 2.x configuration for high load servers (Apache Users)

On 27 May 2011 20:52, Rob Morin <rob@xxxxxxxxxxxx> wrote:

We use a CDN for our images and other static files like CSS and so on…

Like I mentioned keepalive is off as the Hosting company said we cannot use it in conjunction with keepalived on our haproxy setup, I guess I can test it on one machine and see if anything goes awry..

I have not considered compiling it myself yet, I wanted to see how far I can go with respect to tuning before I have to rip down apache and compile it, we do have a fully managed account with the hosting company but I am sure if I ask to compile apache they might disagree with me on that and might not support it… most hosting companies now only support packages that are in the main repos of that distribution and will not vary from that even for compiling..

Yes, updates do tend to get more specific here:)

As for profiling I have asked the Dev guys to see if they can do something like that, however New Relic does help them a bit but nothing like xdebug I guess..

If I am not mistaken, New Relic is metric measured from outside, where Xdebug gives you detailed profile of your code execution path and delays, it is actually very informative just to try it out. At one of the companies I work for we discovered that 80% of page load time the code was just setting object properties (some braindead design by using methods (and subsequently __call) to set every variable, don't mention it:)

Nor sure I understand where you say memcache is slow, we uses memcache for session variables and for mysql query caching… I did not think there was anything as reliable and fast or faster than that?

Can you elaborate on file-based-cache fro mysql querrie? Or point me to some docs?

Unfortunately I don't have any published measurements, but here it goes: we had an application that we benchmarked and if we used memcache for caching mysql queries it gave 100 requests per second tops. Just switching to file based caching gave us around 150-170 requests per seconds. This went well until there were not too many files on the filesystem. This also implies two predispositions:
- you can have independent per-server cache
- you do not cache too many small objects and cache files have to stay in VFS cache (maybe this could be remedied with some reiserfs which excels at small files but it is (was?) not suitable because of big kernel lock)

Why is memcache slower than file cache? Because using memcache causes additional delay by communication between two userland processes (your app server and memcached: form request, send it, memcache parses it, searches itself, forms response, sends it, app receives it and parses it) where VFS brings cache directly from memory (if all fits in there). VFS cache is even faster than ramdisk if all that needs to be cached fits into your ram.

Take care,
b.

We use munin, so we have most of what you are requesting… I will see about making them open to the world for a couple days for you to take a peek at…

Thanks for your reply, its very much appriciated…

Have a great weekend!

Rob Morin

Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207

From: Bostjan Skufca [mailto:bostjan@xxxxxx]
Sent: Friday, May 27, 2011 12:23 PM

To: users@xxxxxxxxxxxxxxxx
Subject: Re: Apache 2.x configuration for high load servers

Hi Rob,

couple of thoughts below, please inform me if I make wrong deductions somewhere, tnx.

1. Your static content is served from elsewhere, you are only talking about application servers
2. Your average response time is 200ms, and given 4cores per server and excluding delays you are currently processing 20 requests per second per server tops.
3. If your load is hitting above number of cores your requests are not served as fast as possible
4. Keepalive timeout, what is your setting? Given the presumption that you don't serve static content yourself it means that you only serve master html - you don't need it at all (with given circumstances of course, where each open connection in keepalive state consumes one apache process)
4a. On the second thought maybe you can use keepalive between load balancer and APP servers, to avoid additional delay of connection setup
5. Did you consider compiling apache yourself? Build it statically with all modules compiled in (and as DSO only those modules you need occasionally) and with -O3? The latter calls for careful testing. Same goes for PHP
6. Did you try profiling your application? Use xdebug to make a request trace and start optimizing there, usually it is far more efficient that system-level optimization.
7. Memcache is slow compared to file-based cache, given that you have enough ram to hold all your files in cache. We've performed the tests and unless you don't need distributed cache filesystem cache is better/faster solution.
8. Maybe latencytop is the next optimization tool, or ftrace if you are using linux.
9. IO graphs (IOPS, bytes per seconds, disk time per second, time per IOP), do you have them?
10. Can you provide also load, cpu utilization, network bytes and packets, and memory usage graphs?

b.

On 27 May 2011 14:24, Rob Morin <rob@xxxxxxxxxxxx> wrote:
Thanks for all your help... first off I increased MaxRequestsPerChild from
1500 to 10000 on one webserver, just as a test to make sure all is good....

As for KeepAlive, we were told we cannot use it due to we are also using
keepalived on our haproxy load balancer, is this true?? As we have a bunch
of SSL certs

I already had ServerToken set to prod :)

I am not sure about UseCanonicalName On, as from what I read it might break
our rewrite rules we use them alott

So I commneted out mod_dir and Apache would not start I got this error, so I
put it back..

Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf:
Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module
not included in the server configuration

I will disable atime later on this morning, I will mod fstab and reboot one
server at a time... the load balancer will autoamtiocally remoe that server
form the cluster when its down so ti will not affect any clients.. :)

I will read up more on mod_file_cache

Thanks again...

Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207

-----Original Message-----
From: Geoff Millikan [mailto:gmillikan@xxxxxxxxxxxxx]
Sent: Friday, May 27, 2011 3:35 AM
To: users@xxxxxxxxxxxxxxxx
Subject: RE: [users@httpd] Apache 2.x configuration for high load servers

#Increase this number. The zombies you're seeing
#is every time an Apache child process dies.
#The child will die @1500. We run at 20000 and are having no problems.
MaxRequestsPerChild 1500

#We don't run as hot as you (we average 3 hits/sec and max at like 15
#in a 24 hour period) but here's our settings:
StartServers 70
MinSpareServers 70
ServerLimit 364
MaxClients 364
MaxRequestsPerChild 20000

#Shrink the size of the response header:
ServerTokens Prod

#Defiantly want keepalive on, it will help page load time for customers.
#Everything else looks good.
Keepalive is on

#We have this turned on but we use server side includes SSI and we cache.
UseCanonicalName On

You have way too many modules enabled. You need user_dir module? Every
module loaded takes up RAM. You should be able to get the
"RES" RAM use per process down to under 20 MB unless you're doing something
funky.

If you're finding that 80% of the time you're running 100 processes, then
start that many and keep that many going. We found that
the process that starts up new children takes too long and it was causing
pages to hang while Apache started up new children. So
just start as many as you need, even if a lot of them are just idle.

Couple of other tips:

1. Turn off the "access time" atime on your file system. This will speed
disk access a lot.

2. Using mod_file_cache on a big site is hard to get everyone on board for.
It's complicated and it's easy to cache the wrong
stuff. However, with a bit of pain, there is a lot of gain to be had there
in terms of reduced work load on the server and thus
faster page load times. Suggest you bite the bullet and do it. BTW, the
CentOS rpm of Apache doesn't work with mod_file_Cahce.
Lots of bugs. So have to compile from source on this. We're on CentOS 5
too.

Lots of people will suggest the Worker MPM due to it's lower RAM usage
however we haven't done it as the PHP web site speaks so
strongly against it.

Best,

Geoff Millikan @ http://www.t1shopper.com/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx