RE: "proxy_balancer" | stickysession

"King Holger (CI/AFP2)" <Holger.King@xxxxxxxxxxxx> · Mon, 25 Oct 2010 15:11:16 +0200

Hi Rainer,

we discovered the same Tomcat-"connection-switch" problem again. This time with the Apache2 ERROR-LOG, containing:

Apache2-ERROR-Log Output:
[Mon Oct 25 10:18:58 2010] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Mon Oct 25 11:46:35 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 25 11:50:02 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

Apache2-ACCESS-Log Output:
10.35.32.123 - - [25/Oct/2010:11:50:04 +0200] "POST /servlet/ClientIO/1ojgw1th835ue/s6/y1 HTTP/1.1" 500 1258 "-" "Jakarta Commons-HttpClient/3.1" "JSESSIONID=A62BCC2B1054CA532C68CA409F41548C.rb-wcmstc1" "A62BCC2B1054CA532C68CA409F41548C.rb-wcmstc1" "JSESSIONID=56DB518EDE4EFED6337349F3478A4863.rb-wcmstc2; Path=/" 2795

When checking the configuration file for the MPM ("httpd-mpm.conf"), the central Apache2-configuration does not include that file:
# Server-pool management (MPM specific)
#Include conf/extra/httpd-mpm.conf

Currently, we use the "worker"-MPM as shown in the "apachectl"-output (AND NOT WORKER):
<hostname>:/opt/wcms/apache/bin $ ./apachectl -M | grep -i "worker"
Syntax OK
 mpm_worker_module (static)

When monitoring the Apache2 via the "/server-status" resource, the following information is shown (see screenshot enclosed):

As you can see:
- the total amount of requests possibly being handled in parallel: 400
- the amount of idle workers sometimes reaches the critical value: 0 (not documented in the screenshot enclosed)

Our questions:
- why 400? We assumed a default value of 150 for "MaxClients" (see "httpd-mpm.conf")
- should we increase the "MaxClients"-value by explicitly defining it in "httpd-mpm.conf" (as mentioned in the "error.log" above)?

When monitoring one of the Tomcats below "/manager/status" (here: "rb-wcmstc2"), the screenshot enclosed shows the following information:
- MaxThreads: 500
- ThreadCount: 252
- Current Thread Busy: 239

The configuration for the Tomcats (here "rb-wcmstc2"):
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" allowTrace="true" maxThreads="500" />

Consequence:
- it seems to be clear, that Apache2 is the bottleneck (the amount of connections available within the "connection-pool" does sporadically reach the value of "0")

What are your suggestions?

Kind regards,
Holger King

-----Original Message-----
From: Rainer Jung [mailto:rainer.jung@xxxxxxxxxxx]
Sent: Freitag, 15. Oktober 2010 15:26
To: users@xxxxxxxxxxxxxxxx
Subject: Re:  "proxy_balancer" | stickysession

On 15.10.2010 12:32, King Holger (CI/AFP2) wrote:
> it's a pity - but the clocks are in sync (using NTP). On both Tomcats and the Apache2 instances.

Good to know.

> According to the following command output:
> grep -i "52C326B80A73EFF19CEE49B013533F06" localhost_access_log.2010-10-14.log
>
> the last pattern match for the JSessionID mentioned below is:
> 10.35.32.123 - - [14/Oct/2010:11:45:01 +0200] POST /servlet/ClientIO/90i8dcztq97l/s6/21i HTTP/1.1 200 197 52C326B80A73EFF19CEE49B013533F06.rb-wcmstc2
>
> So, I cannot find any further request logged further down in Tomcat "rb-wcmstc2".

So we deduce, that either the request is still hanging inside Tomcat or
it never reached Tomcat. Although the latter is more likely, you can
check: if you have the Tomcat manager webapp deployed, you can look at
the status page, which shows a list of all in-flight requests.

> Due to an already overwritten error.log (log-rotation), I do not have any more access to the Apache2 error-log. :(

Then there's likely no way forward for this incident. We would nee dto
wait for the next one :(

> We will add the "%D" to the log format string on both Apache2 and Tomcat.

Good, it's helpful in a lot of situations anyhows.

> Any more hints to identify the problem? The problematik POST request seems to be:
>> 10.35.32.123 - - [14/Oct/2010:11:45:03 +0200] "POST /servlet/ClientIO/90i8dcztq97l/s6/21j HTTP/1.1" 500 1258
>> "-" "Jakarta Commons-HttpClient/3.1" "JSESSIONID=52C326B80A73EFF19CEE49B013533F06.rb-wcmstc2"
>> "52C326B80A73EFF19CEE49B013533F06.rb-wcmstc2" "JSESSIONID=B3C4AABB5F983A0E9D6478C42C88A5C4.rb-wcmstc1; Path=/"
>
> This POST throws a 500 status code.

Sorry, to many possible reasons in the proxy for that. You need the
error log.

You can try to make your setup a bit more robust by looking at the
parameter table given at

http://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypass

For the ajp workers, the following parameters come to mind:

- ping (e.g. ping=10s)
- connectiontimeout (e.g. connectiontimeout=10s)
- timeout (e.g. timeout=120s)
- keepalive=On

The 10 second timeouts could also be lowered to e.g. 5 or 2 seconds if
you are reasonably sure that you don't have any longer GC pauses on the
Java back-ends. A good value for the general timeout depends on your
expectation of response times the back-end will be able to support as
long as it is running well. The "%D" in the access log will help you get
some real numbers.

Regards,

Rainer

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx