Hi, I have two web servers that proxy around 150 backend sites, and are experiencing a lot of mod_proxy timeout errors. The two servers are Xen guests running CentOS 5.3, and sit behind an LVS-NAT load balancer which is also a Xen domU, also running CentOS 5.3. The errors in the error log are: [error] (70007)The timeout specified has expired: proxy: HTTP: attempt to connect to x.x.x.x:80 (www.somesite.com) failed [error] ap_proxy_connect_backend disabling worker for (www.somesite.com) [error] proxy: HTTP: disabled connection for (www.somesite.com) The last error above appears every time a request is made for www.somesite.com until that backend site is re-enabled by mod_proxy. The errors affect a number of different backend sites with the only common factor appearing to be that these are among the most popular reverse-proxied sites. The errors don't seem to appear for the less popular backend sites. Something I've observed a number of times is that a TCP connection to a particular backend site will be stuck in the SYN_SENT state for a long time (approx. 2 minutes), and as soon as that connection disappears the timeout error occurs and the backend site is disabled. Until last week this service was running on a single domU with no load balancer, on completely different hardware. In this configuration the timeout errors had occurred however they were nowhere near as consistent - about 2 or 3 times a month compared to at least once per hour that we're experiencing now. We receive an average of around 300,000 hits per hour, up to about a million during busy parts of the day (as reported by AWStats). We have a 10Mbps internet link however our cacti graphs show the average usage as being around 2Mbps, peaking to 3Mbps, so it's unlikely that we're saturating the network link. Here is an example virtual host directive from our configuration files: -------- <VirtualHost *:443> ServerAdmin admin@xxxxxxxxxxxxxx DocumentRoot /var/www/html ServerName "www-somesite-com.ourcompany.com" LogLevel warn ServerSignature Off SSLEngine on SSLProtocol all -SSLv2 SSLCipherSuite HIGH:+MEDIUM:!EXP:!SSLv2:!LOW SSLCertificateKeyFile /etc/httpd/certs/sp.key SSLCertificateFile /etc/httpd/certs/sp.cer SSLCertificateChainFile /etc/httpd/certs/server-chain.crt ProxyPass / http://www.somesite.com/ ProxyPassReverse / http://www.somesite.com/ ProxyHTMLURLMap http://www.somesite.com / SetOutputFilter line-editor RequestHeader unset Accept-Encoding ProxyPassReverseCookieDomain .somesite.com .ourcompany.com SetEnv LineEdit "application/x-javascript;text/plain;text/css;text/javascript;text/html" LERewriteRule "\"http\://www\.somesite\.com/" "\"/" R LERewriteRule "\'http\://www\.somesite\.com/" "'/" R Include conf.d/shib.conf.include ErrorDocument 502 https://sp.ourcompany.com/errors/providerSiteError_502.html SetEnv proxy-nokeepalive 1 </VirtualHost> -------- Here are some more details of the web server: # uptime 14:04:05 up 5:28, 1 user, load average: 0.00, 0.01, 0.02 # uname -a Linux ckn-sp-2 2.6.18-128.4.1.el5xen #1 SMP Tue Aug 4 20:51:12 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux # free total used free shared buffers cached Mem: 4194304 946272 3248032 0 32312 127636 -/+ buffers/cache: 786324 3407980 Swap: 2031608 0 2031608 # /usr/sbin/httpd -v Server version: Apache/2.2.3 Server built: Jul 14 2009 06:02:39 # netstat -s Ip: 2934756 total packets received 0 forwarded 0 incoming packets discarded 2934742 incoming packets delivered 3207218 requests sent out Icmp: 2267 ICMP messages received 10 input ICMP message failed. ICMP input histogram: destination unreachable: 286 redirects: 1516 echo requests: 465 773 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 308 echo replies: 465 IcmpMsg: InType3: 286 InType5: 1516 InType8: 465 OutType0: 465 OutType3: 308 Tcp: 61435 active connections openings 110304 passive connection openings 36 failed connection attempts 2578 connection resets received 86 connections established 2777064 segments received 3011011 segments send out 40041 segments retransmited 0 bad segments received. 318 resets sent Udp: 157061 packets received 264 packets to unknown port received. 0 packet receive errors 157337 packets sent TcpExt: 3 invalid SYN cookies received 36 resets received for embryonic SYN_RECV sockets 39 packets pruned from receive queue because of socket buffer overrun 2 ICMP packets dropped because socket was locked 95627 TCP sockets finished time wait in fast timer 735 time wait sockets recycled by time stamp 53736 delayed acks sent 28 delayed acks further delayed because of locked socket Quick ack mode was activated 1415 times 37023 packets directly queued to recvmsg prequeue. 1568 packets directly received from backlog 414402 packets directly received from prequeue 835894 packets header predicted 25303 packets header predicted and directly queued to user 641977 acknowledgments not containing data received 491841 predicted acknowledgments 653 times recovered from packet loss due to fast retransmit 129 times recovered from packet loss due to SACK data 11 congestion windows fully recovered TCPDSACKUndo: 39 35 congestion windows recovered after partial ack 44 TCP data loss events 104 timeouts after reno fast retransmit 29 timeouts after SACK recovery 8 timeouts in loss state 1044 fast retransmits 22 forward retransmits 779 retransmits in slow start 5787 other TCP timeouts TCPRenoRecoveryFail: 19 3 sack retransmits failed 1 times receiver scheduled too late for direct processing 1744 packets collapsed in receive queue due to low socket buffer 1227 DSACKs sent for old packets 144 DSACKs received 124 connections reset due to unexpected data 689 connections reset due to early user close 3991 connections aborted due to timeout 10 times unabled to send RST due to no memory IpExt: ---- All ideas are welcome. Thanks, Nick --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx