"proxy_balancer" | stickysession

"King Holger (CI/AFP2)" <Holger.King@xxxxxxxxxxxx> · Wed, 27 Oct 2010 16:10:47 +0200

Hi user mailing list, Hi Rainer,

we need your help :) We re-discovered a Tomcat-connection switch problem in combination with Apache2 2.2.16 and "mod_proxy_balancer". This time, we saved the Apache2 ERROR-LOG!

The Apache2-ACCESS log documents the connection switch from one Tomcat named "rb-wcmstc1" to the other one "rb-wcmstc2":
10.35.32.123 - - [25/Oct/2010:11:50:04 +0200] "POST /servlet/ClientIO/1ojgw1th835ue/s6/y1 HTTP/1.1" 500 1258 "-" "Jakarta Commons-HttpClient/3.1" "JSESSIONID=A62BCC2B1054CA532C68CA409F41548C.rb-wcmstc1" "A62BCC2B1054CA532C68CA409F41548C.rb-wcmstc1" "JSESSIONID=56DB518EDE4EFED6337349F3478A4863.rb-wcmstc2; Path=/" 2795

Apache2-ERROR-Log Output:
[Mon Oct 25 10:18:58 2010] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Mon Oct 25 11:46:35 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 25 11:50:02 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

Our assumption: a bottleneck within the Apache2 connection pool (http://www.gossamer-threads.com/lists/apache/users/343190)???

When monitoring the Apache2 via the "/server-status" resource, the following information is shown (see screenshot enclosed). As you can see:
- the total amount of requests (sum of requests and idle workers) possibly being handled in parallel: 400
- the amount of idle workers sometimes reaches the critical value: 0 (not documented in the screenshot enclosed)

After checking the Apache2 central configuration file for the Apache2-MPM ("httpd-mpm.conf"), it does NOT include that resource:
# Server-pool management (MPM specific)
#Include conf/extra/httpd-mpm.conf

Currently, we use the "worker"-MPM as shown in the "apachectl"-output (AND NOT PREFORK).
<hostname>:/opt/wcms/apache/bin $ ./apachectl -M | grep -i "worker"
Syntax OK
 mpm_worker_module (static)

Our questions:
- when not including the "httpd-mpm.conf" (containg configurations for different MPMs), which values are used?
- why 400 requests? We assumed a default value of 150 for "MaxClients" (see "httpd-mpm.conf")
- should we increase the "MaxClients"-value by explicitly defining it in "httpd-mpm.conf" (as mentioned in the "error.log" above)?

When monitoring one of the Tomcats below "/manager/status" (here: "rb-wcmstc2"), the screenshot enclosed shows the following information:
- MaxThreads: 500
- ThreadCount: 252
- Current Thread Busy: 239

The configuration for the Tomcats (here "rb-wcmstc2"):
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" allowTrace="true" maxThreads="500" />

Consequence:
- it seems to be clear, that Apache2 is the bottleneck (the amount of connections available within the "connection-pool" does sporadically reach the value of "0")

Do you agree? What are your suggestions?

Kind regards,
Holger King

-----Original Message-----
From: Rainer Jung [mailto:rainer.jung@xxxxxxxxxxx]
Sent: Freitag, 15. Oktober 2010 15:26
To: users@xxxxxxxxxxxxxxxx
Subject: Re:  "proxy_balancer" | stickysession

On 15.10.2010 12:32, King Holger (CI/AFP2) wrote:
> it's a pity - but the clocks are in sync (using NTP). On both Tomcats and the Apache2 instances.

Good to know.

> According to the following command output:
> grep -i "52C326B80A73EFF19CEE49B013533F06" localhost_access_log.2010-10-14.log
>
> the last pattern match for the JSessionID mentioned below is:
> 10.35.32.123 - - [14/Oct/2010:11:45:01 +0200] POST /servlet/ClientIO/90i8dcztq97l/s6/21i HTTP/1.1 200 197 52C326B80A73EFF19CEE49B013533F06.rb-wcmstc2
>
> So, I cannot find any further request logged further down in Tomcat "rb-wcmstc2".

So we deduce, that either the request is still hanging inside Tomcat or
it never reached Tomcat. Although the latter is more likely, you can
check: if you have the Tomcat manager webapp deployed, you can look at
the status page, which shows a list of all in-flight requests.

> Due to an already overwritten error.log (log-rotation), I do not have any more access to the Apache2 error-log. :(

Then there's likely no way forward for this incident. We would nee dto
wait for the next one :(

> We will add the "%D" to the log format string on both Apache2 and Tomcat.

Good, it's helpful in a lot of situations anyhows.

> Any more hints to identify the problem? The problematik POST request seems to be:
>> 10.35.32.123 - - [14/Oct/2010:11:45:03 +0200] "POST /servlet/ClientIO/90i8dcztq97l/s6/21j HTTP/1.1" 500 1258
>> "-" "Jakarta Commons-HttpClient/3.1" "JSESSIONID=52C326B80A73EFF19CEE49B013533F06.rb-wcmstc2"
>> "52C326B80A73EFF19CEE49B013533F06.rb-wcmstc2" "JSESSIONID=B3C4AABB5F983A0E9D6478C42C88A5C4.rb-wcmstc1; Path=/"
>
> This POST throws a 500 status code.

Sorry, to many possible reasons in the proxy for that. You need the
error log.

You can try to make your setup a bit more robust by looking at the
parameter table given at

http://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypass

For the ajp workers, the following parameters come to mind:

- ping (e.g. ping=10s)
- connectiontimeout (e.g. connectiontimeout=10s)
- timeout (e.g. timeout=120s)
- keepalive=On

The 10 second timeouts could also be lowered to e.g. 5 or 2 seconds if
you are reasonably sure that you don't have any longer GC pauses on the
Java back-ends. A good value for the general timeout depends on your
expectation of response times the back-end will be able to support as
long as it is running well. The "%D" in the access log will help you get
some real numbers.

Regards,

Rainer

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx