We finally figured out the reason for these 503 errors was an actual connectivity issue to the backend. net.ipv4.tcp_syn_retries kernel parameter on Linux was set as 0 maybe to avoid DDoS attacks this was causing connect to fail on the first try, setting it to the OS default 5 causes OS to send SYN packets 5 times to initiate 3 way handshake before it gives up on connecting & resolves the 503 issues. Not that this actually undermines underlying network issues, but this stops the bleeding immediately therefore interesting value to verify when looking into similar issues. On Fri, Jul 16, 2010 at 4:01 AM, alin vasile <alinachegalati@xxxxxxxxx> wrote: > Hi, > > Since in your apache logs you can see a timeout while connecting to the > backend : > > "[Wed Jul 14 15:59:46 2010] [error] (110)Connection timed out: proxy: > HTTP: attempt to connect to 1xx.7x.xxx.xxx:80 (web.mycompany.net) > failed" > > it is normal that there not to have any logs at that time. > > The problem here is that your backend may not support a spike load or can > hove some capacity problems. > > ________________________________ > From: pratyoosh sharma <pratyoosh.sharma@xxxxxxxxx> > To: users@xxxxxxxxxxxxxxxx > Sent: Fri, July 16, 2010 7:10:34 AM > Subject: Apache 2.2.11 mod_proxy returns 503 (service > unavailable) under load > > We are running a high volume portal that uses Apache 2.2.11 mod_proxy > to reverse proxy content from downstream apache-> application servers, > we recently started observing 503 errors on the Apache which happen > sporadically throughout the day during activity on random requests. > > The requests returned with 503 don't appear on the backend apache > access logs at all, there are no 503 failures on backend > apache.Network traces of connectivity between frontend apache & the > backend don't reveal anything either. > > An interesting pattern is the failed requests always take ~3-4 seconds > > Frontend Apache: > Error log: > [Wed Jul 14 15:59:46 2010] [error] (110)Connection timed out: proxy: > HTTP: attempt to connect to 1xx.7x.xxx.xxx:80 (web.mycompany.net) > failed > [Wed Jul 14 15:59:46 2010] [error] ap_proxy_connect_backend disabling > worker for (web.mycompany.net) > Access log: > "USER13" xx.xx.xxx.xxx - [14/Jul/2010:15:59:43 -0400] "GET > /myapp/web/css/ext-all.css HTTP/1.1" 503 1030 3041412 > > Backend: > This request never reached the backend server & doesn't shows up in > the access logs of backend server > > > All the failing requests like the above take ~3 seconds so i guess its > like a timeout but i can't find any such configuration. > > Is there some known Apache bug for this behavior? We have retry=0 & > disablereuse on for all proxy pass directives > > What kind of starvation should i look for in Apache we have mod_status > enabled, file descriptor limit set a 64K, CPU is almost always idle & > 25 Gig memory free on the server. > > Thanks > > --------------------------------------------------------------------- > The official User-To-User support forum of the Apache HTTP Server Project. > See http://httpd.apache.org/userslist.html> for more info. > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx > " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx > > > -- Thanks & Regards, Pratyoosh Sharma --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|