Re: Load balancing question (Apache Users)

On 9/17/07, James Sherwood <jsherwood@xxxxxxxxxxxxxxxx> wrote:

Hello,

CORRECTED(status page working now)

We upgraded to the latest mod_jk and this were the results:

1: All monitors were fine, there were no bottlenecks anywhere that we could find(cpu's,HD's and networks all seemed fine).
2: This time when we brought the servers to their knees, they recovered a short time after the test was completed.
3: We tried the socket_keepalive=true for the workers and the server did not recover after
4: the only problem we can find is after the test in the mod_jk log we have about 20-30 lines of this:
[Mon Sep 17 08:03:49.906 2007] [7948:4868] [error] jk_ajp_common.c (2097): (tomcat5) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port

The lines vary only by the (tomcat5) being any of the tomcats in the loadbalance.

It seems like apache/tomcat/mod_jk are reaching the max number of connections between each other or something?

Any help would be GREATLY appreciated,
--James

----- Original Message -----
From: James Sherwood
To: users@xxxxxxxxxxxxxxxx

Sent: Monday, September 17, 2007 9:12 AM

Subject: Re: Load balancing question

Hello,

I cannot get my mod_jk status page to work. Maybe it is because I am on windows?

It seems:

worker.list=jk-manage
worker.jk-manage.type=status
worker.jk-manage.mount=/admin/status/jk
only takes a linux style path for the mount?

We upgraded to the latest mod_jk and this were the results:

1: All monitors were fine, there were no bottlenecks anywhere that we could find(cpu's,HD's and networks all seemed fine).
2: This time when we brought the servers to their knees, they recovered a short time after the test was completed.
3: We tried the socket_keepalive=true for the workers and the server did not recover after
4: the only problem we can find is after the test in the mod_jk log we have about 20-30 lines of this:
[Mon Sep 17 08:03:49.906 2007] [7948:4868] [error] jk_ajp_common.c (2097): (tomcat5) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port

The lines vary only by the (tomcat5) being any of the tomcats in the loadbalance.

It seems like apache/tomcat/mod_jk are reaching the max number of connections between each other or something?

Any help would be GREATLY appreciated,
--James

----- Original Message -----
From: Bj
To: users@xxxxxxxxxxxxxxxx
Sent: Saturday, September 15, 2007 5:17 AM
Subject: Re: Load balancing question

What says your mod_jk status page ?
try to monitor during the load to see if your workers are in error or OK state, il the max busy is reached,....
Then look at your logs (mod_jk, apache, tomcat, webapps logs, windows,...)

As said before, you should check the number of tcp connections opened. If your do not use keep alive feature you can have a bootleneck there (apache and tomcat servers).You can also have error like max opened file reached.
Then look at the load average,system cpu, iowait,..

You can also monitor your tomcats through JMX (using jconsole or missioncontrol) to check that garbage collections works fine and just don't hang up too long.

try to deactivate the 2 tomcat instances on your apache server to see if httpd is still available after the load test.

--
Bj

On 9/14/07, James Sherwood <jsherwood@xxxxxxxxxxxxxxxx> wrote:
Hello,

Everything is Windows2003 Server.

After the load we cannot load pages either through apache or by contacting
tomcat directly.

I beleive you are on the right path tho, about connections not getting
released, thats what I figure it is too but I do not know how to fix it.

Thanks,
James

----- Original Message -----
From: "AFrieze" < AFrieze@xxxxxxxxxxxx >
To: <users@xxxxxxxxxxxxxxxx>
Sent: Friday, September 14, 2007 12:02 PM
Subject: Re: Load balancing question

>
>>
>> We also have the problem of once the load stops, the sites are still down
>> but Apache/tomcats still seem to be running fine.  A restart of
>> either(not even both) fixes the sites.
> A guess
>
>  Your apache server is not releasing connections.  If you are running
> linux, type  "netstat -vat" into a terminal on your apache machine, before
> and after you hit your server.  See if the connections are being released.
>
> You could also try typing "ps -e | grep "httpd"" to see how many apache
> processes are being run before/after.  Look in the apache error log, etc.
> You might find a clue like "MaxClients reached"
>
> Question
> Are you able to log into all your tomcats(through port 8080) independent
> of apache and get served requests?  Can you log onto apache and get a
> statically served page?
>
> Cheers
> AFrieze
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL: http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
>
>
> __________ NOD32 2529 (20070913) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com
>
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL: http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

__________ NOD32 2534 (20070917) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com