Phil, Thanks for your response, and all the excellent suggestions. I found a bug issue raised about this very problem https://issues.apache.org/bugzilla/show_bug.cgi?id=47207 In my case I intentionally invoked the 500 error on the backend tomcat as I have seen these issues in the past, so I was hoping to be able to handle this at the apache (load-balance) layer. However, it doesn't sound like this is possible at the moment. Thanks again, Ed. -----Original Message----- From: Philip Wigg [mailto:phil@xxxxxxxxxxxxxxxx] Sent: 10 March 2010 09:28 To: users@xxxxxxxxxxxxxxxx Subject: Re: RE: mod_proxy_balancer back end status check On 9 March 2010 23:01, Edward Quick <Edward.Quick@xxxxxxxxxxxxx> wrote: > Further testing showed this succeeded when the server was busy. The balancer marked the member's status as Err. > However later after firing continuous requests, the backend ran out of memory, causing it to return a 500, and then the balancer marked it up again, sigh! > > I'm considering writing a script to check the backend and disable the members via a curl call to the balancer-manager page, but that seems a bit crude. Is there anything else I can add to the configuration below which might handle a code 500? Hmm. Unfortunately I don't know of anything else you could do to fix that with mod_proxy. Someone else might have a way. Are you sure the 500 is definitely being received from the back-end and not from Apache/mod_proxy itself because it doesn't have anywhere left to route requests to? I'm surprised Tomcat returns a cpong in that scenario but I've never tried it. Presuming it is Tomcat returning the 500, you might try:- 1. Make sure it is running out of heap that's taking down Tomcat and not something else like running out of PermGen space or threads. Presuming you've done this... 2. Fixing the app so it doesn't run out of heap. I appreciate if you're the sysadmin that it can be difficult to get the devs to do this. 2. Is the app in production currently? How long does it typically take to run out of heap space? Can you just restart it once a day whilst you encourage the devs to fix it? If not, can you increase the max heap space until it lasts longer than a day? You can get stats from Tomcat in XML format by accessing the manager page using /manager/status?XML=true which makes it easy to log stats you're interested in using curl, grep, or perl. 3. Monitor your log files for JavaOutOfMemory errors (you should get these if Tomcat is running out of heap) using something like http://sourceforge.net/projects/swatch/ which allows you to take an action if it sees that string. Use it to automatically restart Tomcat in the meantime (whilst the devs are fixing the main issue). I've used this successfully in the past when all else fails. 4. Do you only have one front-end web server? If you have multiple web servers, do you have a hardware load-balancer in front? If so, tie each Apache server to a single Tomcat instance and use the front-end load-balancer to remove the web server from the load-balanced pool once it starts throwing a 500. You'll need to make sure you always have at least one working server Tomcat though by implementing something like option 3. The Tomcat mailing list would be best placed to answer questions about going down the fixing-Tomcat road. Good luck fixing your issue. -- Phil --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc.) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Index Ltd is a company registered in England and Wales under number 01190902. VAT registration number 761 2978 07. Registered Office: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and regulated by the Financial Services Authority. FSA Register number 114059. --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|