Intermittent stalls

[Jeroen Van den Horn <jeroen@xxxxxxxxxxxx>]

I hope somebody can help me with an issue that's kept me busy for quite some days now.

We are using LVS in combination with Apache and mod-proxy to serve content from a large number of back-end servers (primarily using mod_proxy_balancer and mod_proxy_ajp).

All runs smoothly until one of the back-end servers start behaving eratically (i.e. takes a long time to respond). When this occurs the apache server seems to stall - that is, after some time it seems to get stuck waiting for the request to time-out. In the intermediate time though it is still serving requests just fine. I've tried the worker, prefork and event MPM and all experience this same behavior.

Could it be that something like this is happening:

The 'master' apache process that dispatches incoming connections to the underlying servers allocates a 'slot' to process a request (for the sake of the argument, call this slot #10). Further requests are received and allocated to higher numbered slots. This goes fine until the list wraps and at a certain point we arrive back at slot #10. Now for some obscure reason apache appears to somehow sit-and-wait until this connection either times-out or deliveres the data. As soon as than happens everything just runs fine again, but during this intermittent time the server just appears to sit and do nothing (and appears stalled). At this moment we cannot even serve local content (which - as a side effect causes the LVS director to box the node off-line).

Has anybody ever seen Apache behaving like this? Could it be something to do with some module that we use? We use mod_perl quite extensively (and that's where my gut feelings point to but before rewrite all our code in a native mod_xxx module I would like to have just a little more evidence). 

The apache version we are running is:

Server version: Apache/2.2.17 (Ubuntu)

Server built:   Feb 14 2012 19:59:03

Any hints on how to pursue this problem further are much appreciated.

Jeroen