Hi there, we're having a strange issue and wondering if anyone might have any suggestions. We have a cluster of apache machines and are experiencing a problem where the load average and other hardware metrics are well below normal ranges, but the response times are exceedingly high (even for tiny pieces of static content transferred over localhost). We've been running Apache in production for several years now without any issues, and this problem just started yesterday. This likely means the problem isn't directly caused by Apache but is affecting it nonetheless. If we take an httpd server out of the pool via our load balancer, the response times go back to normal. Here's an example of the behavior I'm describing. [root@RHL060 tmp]# time wget http://apc.rhl060.in.xxxxxxx.com/foo.html --2010-08-31 15:49:43-- http://apc.rhl060.in.xxxxxxx.com/foo.html Resolving apc.rhl060.in.xxxxxxx.com... 127.0.0.1 Connecting to apc.rhl060.in.xxxxxxx.com|127.0.0.1|:80... failed: Connection timed out. Retrying. --2010-08-31 15:52:53-- (try: 2) http://apc.rhl060.in.xxxxxxx.com/foo.html Connecting to apc.rhl060.in.xxxxxxx.com|127.0.0.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4 [text/html] Saving to: `foo.html.72' 2010-08-31 15:53:14 (558 KB/s) - `foo.html.72' saved [4/4] real 3m31.004s user 0m0.000s sys 0m0.003s For the problem illustrated above, the loopback has no iptables rules applied. Running `netstat -tapn | grep 80 | wc -l' shows 11299 connections to port 80. We're setup to use prefork with the following configuration. <IfModule prefork.c> StartServers 100 MinSpareServers 100 MaxSpareServers 200 ServerLimit 1000 MaxClients 1000 MaxRequestsPerChild 100000 </IfModule> # uptime 16:06:47 up 14 days, 17:47, 5 users, load average: 2.45, 2.54, 2.74 Checking iostat shows very little disk utilization. The server is running CentOS 5.5 with kernel version 2.6.18-194.8.1.el5. This particular server has 24 cores and 50G of RAM. A number of our servers are experiencing the same issue, and they all have similar hardware and software specs. I'm pretty stumped here. Any help would be highly appreciated. -Travis --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|