Apache2 and Tomcat load balancing | Timeouts: request, connection, cping/cpong

["King Holger (CI/AFP2)" <Holger.King@xxxxxxxxxxxx>]

Dear Apache2 community,

 

having installed Apache2 2.4.3 in combination with:

• mod_proxy_balancer (not “mod_jk”!)
• mod_proxy_ajp
• mod_proxy

and the following virtual host configuration content:

 

<VirtualHost *:8000>
…

BalancerMember ajp://rb-wcmstc1.xx.xxxxx.xxx:8010 loadfactor=50 ping=5 retry=10 route=rb-wcmstc1 connectiontimeout=5 timeout=300
BalancerMember ajp://rb-wcmstc2.xx.xxxxx.xxx:8010 loadfactor=50 ping=5 retry=10 route=rb-wcmstc2 connectiontimeout=5 timeout=300

…
<VirtualHost>

 

to allow and setup a Tomcat 6 load balancing infrastructure, we regularly see the above configured timeouts in seconds (see http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass):

- request-timeout (300 seconds)
- connection-timeout (5 seconds)
- cping/cpong-timeout (5 seconds)

being exceeded. The Apache2 error log proves that:

1. [Mon Apr 08 18:23:16.954749 2013] [proxy_ajp:error] [pid 5030:tid 48] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
2. [Mon Apr 08 18:25:28.422599 2013] [proxy_ajp:error] [pid 6623:tid 52] (70007)The timeout specified has expired: [client 10.25.131.11:28706] AH00897: cping/cpong failed to xx.xx.xx.xxx:8010 (rb-wcmstc1.xx.xxxxx.xxx), referer: https://rb-xxxxx.xx.xxxxx.xxx/fs4webedit/message.jsp

1. might be logged when a request took longer >300 seconds
2. might be logged when ping request took longer >5 seconds

Taking a look at the HTTPS Access logs, we see for:

1. a request really taking 300 sec. It leads to the above error log entry at 18:23:16 – five minutes later:
   xx.xx.xx.xxx - - [08/Apr/2013:18:18:16 +0200] "POST /fs4webedit/Dispatcher HTTP/1.1" 302 295 "…" "F0FDAFA023B323A62813598C985C3AB1.rb-wcmstc1" "-" TLSv1 RC4-MD5 300004793 

With an important BUG fix provided since Apache2 version 2.2.22 we have a solution in place that prevents marking all AJP-workers (used for communication between Apache2 and Tomcat) pointing to the Tomcat instance in ERROR state where the single long running request occurred. This bugfix only works when setting the “ping” attribute for the BalancerMember (see above). Details see ChangeLog enclosed: http://www.apache.org/dist/httpd/CHANGES_2.2.22 respectively http://mail-archives.apache.org/mod_mbox/httpd-cvs/201201.mbox/%3C20120104201529.54EA823888D2@xxxxxxxxxxxxxxx%3E

 

Now, our questions are:

• what could provoke the 2^nd error message: “AH01012: ajp_handle_cping_cpong: ajp_ilink_receive failed” (a stop-the-world garbage collection on Tomcat side?)
• what are the consequences if that happens?
• how to prevent it (simply increasing the ping timeout?)

Best regards,

Holger King