When we were just using Tomcat in our local development environment, we encountered no issues with operations that require large page load times and a large number of HTTP requests.
However, once we added Apache HTTP server to the mix in our hosted QA environment, we encountered errors that point to the mod_proxy_ajp module not correctly closing its connections from Apache to Tomcat (ProxyPass) and vice versa (ReverseProxyPass), most likely as a result of our misconfiguration. I found errors in the Apache logs that would lead one to believe that this is the case, and found an Apache HTTPD bug (since resolved in a version before ours) that looks like it could have been related at one time:
Possibly related Apache HTTP bug:
---------------
https://issues.apache.org/bugzilla/show_bug.cgi?id=38227
HTTPD error_log:
---------------
[Tue Aug 31 16:23:09 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Aug 31 16:23:09 2010] [error] (120006)APR does not understand this error
code: proxy: read response failed from (null) (localhost)
Tomcat log:
----------
Can't find the exact stack trace at the moment, but it said too many open files.
Then we updated /etc/security/limit.conf to change the default ulimit from 1024 to a soft limit of around 15k and a hard limit of around 30k for all users, and we decreased the Timeout in httpd.conf to 15 seconds.
But this then led to HTTP 503 Service Unavailable errors, so we've moved the timeout up to 90 seconds.
Basically, we just don't know enough about the appropriate Apache / Tomcat configuration to strike the right balance between responsiveness and allowing large requests are a very large number of requests to finish. And we were hoping that someone on this user list might have some experience with configuring Apache, its prefork module, and mod_proxy_ajp that could point us in the right direction.
Thanks in advance for any advice you can provide. Below is our current relevant Apache and Tomcat configuration:
httpd.conf:
----------
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 90
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 0
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 3
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 20
MinSpareServers 40
MaxSpareServers 100
ServerLimit 512
MaxClients 512
MaxRequestsPerChild 0
</IfModule>
conf.d/ssl.conf:
---------------
SSLSessionCacheTimeout 600
conf.d/proxy_ajp.conf:
---------------------
# Apache recommends to turn the ProxyRequests directive off
# when using ProxyPass
ProxyRequests off
ProxyPass /images ajp://localhost:8181/images
ProxyPass /rt ajp://localhost:8181/rt
#
# Allows Apache to adjust the URL in the Location, Content-Location and
# URI headers on HTTP redirect responses.
#
ProxyPassReverse /images/ ajp://localhost:8181/images
ProxyPassReverse /rt ajp://localhost:8181/rt
${CATALINA_HOME}/conf/server.xml:
--------------------------------
<!--APR library loader. Documentation at /docs/apr.html -->
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Connector port="8181" enableLookups="false" protocol="AJP/1.3" redirectPort="8282" URIEncoding="UTF-8" connectionTimeout="90000"/>
|