Ya sorry about the attachments... I put URLs to my personal website rather than the work related one to be safe :) http://www.stupidguytalk.org/WEB3.ireel.com-apache_processes-day.png http://www.stupidguytalk.org/WEB3.ireel.com-apache_accesses-day.png Sorry about that.... Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -----Original Message----- From: Rob Morin [mailto:rob@xxxxxxxxxxxx] Sent: Wednesday, May 25, 2011 3:00 PM To: users@xxxxxxxxxxxxxxxx Subject: RE: Apache 2.x configuration for high load servers Thanks for the prompt reply Mark, I will try to add a few things and clear up some stuff I might have omitted in my first post... I have server-status enabled as well as Munin for Apache.. I have attached a munin graphs to this email, not sure if its proper or not... sorry if not... one is accesses and one is processes per day.. To be safe its about 120 accesses per second , each server is basically the same give or take 5%, I will enable access logging for one domain and see what happens with respect to the time to serve a request... Here is a paste of server status for the same server as the attached pngs of munin.. our busy time will start in the next couple hours.... Current Time: Wednesday, 25-May-2011 13:48:42 EDT Restart Time: Sunday, 22-May-2011 04:02:05 EDT Parent Server Generation: 1 Server uptime: 3 days 9 hours 46 minutes 37 seconds Total accesses: 24212224 - Total Traffic: 7.1 GB CPU Usage: u1544.11 s494.57 cu0 cs0 - .692% CPU load 82.2 requests/sec - 25.5 kB/second - 316 B/request 46 requests currently being processed, 384 idle workers _C____._..____._..__C___..__C___.___.____._.._.__._._._..____.__ __C_._..____._.__.____________.___._______________.___._.__.R.__ ._C____CC___C_.__CCR.._._..WW_____.._.C_..C..__.._.__.._______._ __._..__._.__.____.C__C._____..__.__C.____C___._______.C______._ .C_._._.__.________C._____.._..__._..__.C_C__.._.___.__.._____C_ _____._._C__.________.._____.__R_.____.._C_.C.__.CCC.______.__._ ._C_____....__.___..__.____.__.__.____._____...____._._._.__.__C ._C______..______._._._________._.___..____....CC_.__C_C_.C__._. R__...C.__._.___..._.___C_______....__._____._____._____..._.._C __W.__.__.________..___. So then my current config I have seems good enough then, from what you have spoken about so far? StartServers 150 MinSpareServers 125 MaxSpareServers 250 ServerLimit 600 MaxClients 600 MaxRequestsPerChild 1500 As the load at busy times is about 9, and this is for quad core, so there might be room for improvement somewhere I guess.. This is why I turned off logging as we have about 60 domains all writing logfiles via rotatelogs I fogured removing that extra file open and writing to would help a bit... I also made a small RAMdisk to store the eAccelerator cache files rather than use disk... we also use memcache to store sessions and for mysql querries.. We also use Edgecast as our CDN for static content, like images and CSS files.. We also use New Relic and new PHP performance tuning tool, its very cool... that shows that a good 95% of our requests take under 200 ms while the rest can vary between 600 ms and way up to 10 seconds, the 10 second ones are where we connect to processors to valaidate information on clients.. I will read over that in the URL your pasted and see what I can do to still increase performance... As for the logs just to clarify how I did it, all the virtual host config files have the CustomLog entries commented out, with this in minf that means Apache will send access log detaisl to whatever the defautl access log file is , its normally something like /var/log/hhpd/access_log, so in the httpd.con file under CustomLog combined , I just have %h, then in the location for customlog rather than /var/log/httpd/access_log I have /dev/null so this way it will og minimum info and then not write it to disk and send it out to the bit bucket.... we do nto need logs as we use Google Analytics in our code. Is this ok, as if I just remove or comment out the access_log loacation or have no combined entry apache will nto restart.... if I remove mod_log then I will get no errors logs, and those are more important than access_logs for us... Hope this sheds more light on my problem... Thanks again for all your help... Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -----Original Message----- From: Mark Montague [mailto:mark@xxxxxxxxxxx] Sent: Wednesday, May 25, 2011 12:06 PM To: users@xxxxxxxxxxxxxxxx Cc: Rob Morin Subject: Re: Apache 2.x configuration for high load servers On May 25, 2011 10:27 , Rob Morin <rob@xxxxxxxxxxxx> wrote: > We have recently had a 30% increase in traffic and will be expecting > more. I would like to know the correct way to calculate the proper > settings for prefork.c my settings, here is what I have now... > I am going to add more RAM today to make a total of 24 gigs on each > server, and i am going to add an additional web server in the next > week or so... > > Each day we get about 5 million visits... > The settings you use for the prefork MPM are not determined by how many visits you get per day, but by how many concurrent requests the server in question needs to handle during your busiest periods. Number of concurrent requests is in turn determined by how quickly requests come in together with how long it takes to serve (process, respond to, handle) each request. For example, let's say that 90% of the requests you serve require 1 second to serve, but 10% of the requests take 5 seconds to serve. Let's also assume that during your busiest period you are getting 100 requests per second. In each second, then, 100 children will be needed to server these requests -- 90 of these children will finish (and become available to serve another request) after 1 second and the remainder will become available again after 5 seconds. This means that after 5 seconds of your busiest anticipated load, 140 children will be tied up serving requests, and hence you'd want to set ServerLimit comfortably above this value (to allow margin for error and future growth), assuming that you have the necessary hardware resources (RAM, processor cores) to effectively do so. In addition, you may want to set MinSpareServers high enough to handle a spike in load equivalent to one second's worth of requests, and set MaxSpareServers to free up resources if more than the number of children required to server two seconds worth of requests are sitting idle. Access logs with a custom log format containing %D (time required to serve the request in microseconds) are your best tool for figuring out the request patterns that your web server needs to be able to handle. Also, mod_status is a useful tool for seeing what each child process is doing at single point in time (which children are serving requests, which children are idle, and so on). > Apache is currently using 6207 Megs of memory... > > There are 245 Apache processes running at this time. > > -------------------------------- > > So the load is high, but i have free RAM left and only running 245 > Apache processes, out of a possible 600? I only show the first few > lines of top .. > This is anticipated. Apache HTTP Server's prefork MPM will try to keep the number spare processes (children that are idle and waiting for new requests) between MinSpareServers and MaxSpareServers at all times. The rule of thumb I recommend is that you want your system load to be less than twice the number of processor cores. Other people's advice may differ from this, so I recommend that you ask around. If your system load is more than twice the number of processor cores, I recommend: - Determine if there is any storage (disk, etc.) I/O bottleneck; if so, eliminate it. - Determine if there is any network I/O bottleneck (network link, load balancer, firewall, etc.); if so, eliminate it. - Reduce the time required to serve each requests: -- Reduce time required by any web applications run by the web server (active content) -- Reduce time required by the web server itself (web server configuration, operating system configuration) The above recommendations may result in you adding more I/O capacity, more RAM, more processor cores, and/or more machines to your web server cluster. However, I believe it is generally better to only add more resources once you have determined that all existing resources are being used as optimally as possible: improve any web applictions or other active content on your web site so that they require less resources; change your infrastructure architecture (what each web server does and how it does it including caching and proxying); and improve the configuration or your web server (and other servers, including any database servers) and your operating system. > Or do I get this wrong, does each connection have an httpd process > that is run, or are child/subsequent connections, not listed on under > a ps ax?? > If you are using the prefork MPM, a number of child processes get created to handle web server requests (connections). The number of these will vary dynamically based on how you configure the prefork MPM and also on how many requests come in how quickly. All of the processes will show up under ps. > I removed access logging, I disable access logging by commenting out > the CustomLog directive in each virtualhost and then in httpd.com I > set CustomLog value to /dev/null, this way I can keep error working > too. is this ok? Here aer some of the httpd.con entries. > Are you sure that you want to not log web server accesses? This removes an incredibly important diagnostic tool. Also note that if you use CustomLog with a file of /dev/null, then you are having Apache HTTP Server actually create each log entry and send it to the kernel, which will then discard it instead of writing it to disk; if you really want to turn off access logging, then it would be better to completely remove any CustomLog or TransferLog directives from your web server configuration. If you want to improve performance, see the recommendations at http://httpd.apache.org/docs/2.2/misc/perf-tuning.html > Here are loaded modules, I am not sure if I need them all, can I > safely turn any off? You can and should turn off any modules you are not using; this will save some memory and may make your web server a little more secure. You will need to look at what functionality your web site currently relies upon, together with your web server configuration, in order to determine which modules you can avoid loading. -- Mark Montague mark@xxxxxxxxxxx --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|