On 9 March 2010 23:01, Edward Quick <Edward.Quick@xxxxxxxxxxxxx> wrote: > Further testing showed this succeeded when the server was busy. The balancer marked the member's status as Err. > However later after firing continuous requests, the backend ran out of memory, causing it to return a 500, and then the balancer marked it up again, sigh! > > I'm considering writing a script to check the backend and disable the members via a curl call to the balancer-manager page, but that seems a bit crude. Is there anything else I can add to the configuration below which might handle a code 500? Hmm. Unfortunately I don't know of anything else you could do to fix that with mod_proxy. Someone else might have a way. Are you sure the 500 is definitely being received from the back-end and not from Apache/mod_proxy itself because it doesn't have anywhere left to route requests to? I'm surprised Tomcat returns a cpong in that scenario but I've never tried it. Presuming it is Tomcat returning the 500, you might try:- 1. Make sure it is running out of heap that's taking down Tomcat and not something else like running out of PermGen space or threads. Presuming you've done this... 2. Fixing the app so it doesn't run out of heap. I appreciate if you're the sysadmin that it can be difficult to get the devs to do this. 2. Is the app in production currently? How long does it typically take to run out of heap space? Can you just restart it once a day whilst you encourage the devs to fix it? If not, can you increase the max heap space until it lasts longer than a day? You can get stats from Tomcat in XML format by accessing the manager page using /manager/status?XML=true which makes it easy to log stats you're interested in using curl, grep, or perl. 3. Monitor your log files for JavaOutOfMemory errors (you should get these if Tomcat is running out of heap) using something like http://sourceforge.net/projects/swatch/ which allows you to take an action if it sees that string. Use it to automatically restart Tomcat in the meantime (whilst the devs are fixing the main issue). I've used this successfully in the past when all else fails. 4. Do you only have one front-end web server? If you have multiple web servers, do you have a hardware load-balancer in front? If so, tie each Apache server to a single Tomcat instance and use the front-end load-balancer to remove the web server from the load-balanced pool once it starts throwing a 500. You'll need to make sure you always have at least one working server Tomcat though by implementing something like option 3. The Tomcat mailing list would be best placed to answer questions about going down the fixing-Tomcat road. Good luck fixing your issue. -- Phil --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|