Apache2 and Tomcat load balancing | Timeouts: request, connection, cping/cpong

["King Holger (CI/AFP2)" <Holger.King@xxxxxxxxxxxx>]

Dear Apache2 community,

 

having installed Apache2 2.4.3 in combination with:

• mod_proxy_balancer (not “mod_jk”!)
• mod_proxy_ajp
• mod_proxy

 

and the following virtual host configuration:

 

BalancerMember ajp://rb-wcmstc1.de.bosch.com:8010 loadfactor=50 ping=5 retry=10 route=rb-wcmstc1 connectiontimeout=5 timeout=300
BalancerMember ajp://rb-wcmstc2.de.bosch.com:8010 loadfactor=50 ping=5 retry=10 route=rb-wcmstc2 connectiontimeout=5 timeout=300

 

to allow and setup a Tomcat 6 load balancing infrastructure, we regularly see the above configured timeouts in seconds (see http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass):

- request-timeout (300 seconds)
- connection-timeout (5 seconds)
- cping/cpong-timeout (5 seconds)

being exceeded. The Apache2 error logs prove that:

 

1. [Mon Apr 08 18:23:16.954749 2013] [proxy_ajp:error] [pid 5030:tid 48] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
2. [Mon Apr 08 18:25:28.422599 2013] [proxy_ajp:error] [pid 6623:tid 52] (70007)The timeout specified has expired: [client 10.25.131.11:28706] AH00897: cping/cpong failed to xx.xx.xx.xxx:8010 (rb-wcmstc1.xx.xxxxx.xxx), referer: https://rb-xxxxx.xx.xxxxx.xxx/fs4webedit/message.jsp
   

1. might be logged when a request took longer >300 seconds
2. might be logged when ping request took longer >5 seconds

 

Taking a look at the HTTPS Access logs, we see for:

1. a request really taking 300 sec. It leads to the above error log entry at 18:23:16 – five minutes later:
   xx.xx.xx.xxx - - [08/Apr/2013:18:18:16 +0200] "POST /fs4webedit/Dispatcher HTTP/1.1" 302 295 "…" "F0FDAFA023B323A62813598C985C3AB1.rb-wcmstc1" "-" TLSv1 RC4-MD5 300004793

 

 

With an important BUG fix provided since Apache2 version 2.2.22 we have a solution that prevents marking a complete AJP-worker (used for communication between Apache2 and Tomcat) in ERROR state when a single long running request occurred. This bugfix only works when setting the “ping” attribute for the BalancerMember. Details see ChangeLog enclosed: http://www.apache.org/dist/httpd/CHANGES_2.2.22 respectively http://mail-archives.apache.org/mod_mbox/httpd-cvs/201201.mbox/%3C20120104201529.54EA823888D2@xxxxxxxxxxxxxxx%3E

 

Now, our questions are:

• what provokes the 2^nd error message: “AH01012: ajp_handle_cping_cpong: ajp_ilink_receive failed” (a stop-the-world garbage collection on Tomcat side?)
• what are the consequences if that happens?
• how to prevent it (simply increasing the ping timeout?)

 

Mit freundlichen Grüßen / Best regards,

Holger King

CI/AFP2

 

Tel. 0711-811-59112

 

BeQIK