Help: Apache crashed once Concurrent Users reached 150

[Jayaram Ponnusamy <jayaram.ponnusamy@xxxxxxxxx>]

Hello,

We are using apache 2.2.21 on RHEL6.9 and in Backend we are using Tomcat AppServer with JavaBased CMS.

Our Server Architecture is (F5 [For Load Balance] -> Apache WebServer [For Redirection & URL Mapping] -> Tomcat AppServer [Running a Java Based CMS]). 

We are using Apache to connect Tomcat by MOD_JK and mapping the URL (Attached VirtualHost.conf), and No applications/code are running on WebServers.

1. Two WebServers (4 CPU & 8GB RAM on Each Servers)

2. Two TomcatServers (8 CPU & 16GB RAM on Each Servers )

PROBLEM:
When the Child Process count is reached / crossed 200 like below then site is not accessible when the child processes reached 300 or More than site crashed and then we bring the site back by restarting Apache.

When the Child Processes reaching 200 during the time Google Analytics shows the ACTIVE/CONCURRENT Users are 140-160. We have two Apache & Tomcat Servers with Load-balancing but its unable to serve more than 150 concurrent users. Kindly please check my configuration and help to allow at-least 500 concurrent users.

CONFIGURATION:

 httpd.conf

<IfModule prefork.c>
StartServers         80
ServerLimit 3500
MaxClients 3500
MaxRequestsPerChild  0
</IfModule>

Tomcat:
<Connector port="9090" protocol="HTTP/1.1" redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" maxThreads="600" minSpareThreads="10" connectionTimeout="-1" />
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" connectionTimeout="20000" acceptCount="2000" />

http_error.log:
[Thu Dec 05 11:05:13 2019] [info] [client 0.0.0.0] (32)Broken pipe: core_output_filter: writing data to the network
[Thu Dec 05 11:05:13 2019] [info] [client ::1] (32)Broken pipe: core_output_filter: writing data to the network
[Thu Dec 05 11:06:47 2019] [error] [client 0.0.0.0] (70007)The timeout specified has expired: proxy: error reading status line from remote server stg.test.com
[Thu Dec 05 11:06:47 2019] [error] [client 0.0.0.0] proxy: Error reading from remote server returned by /research
[Thu Dec 05 11:06:47 2019] [info] [client 0.0.0.0] (32)Broken pipe: core_output_filter: writing data to the network
[Thu Dec 05 11:06:49 2019] [info] [client ::1] (32)Broken pipe: core_output_filter: writing data to the network
[Thu Dec 05 11:07:02 2019] [info] [client 0.0.0.0] (32)Broken pipe: core_output_filter: writing data to the network

error_log
[Wed Dec 04 14:24:24 2019] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 165 total children
[Wed Dec 04 14:24:25 2019] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 197 total children
[Wed Dec 04 14:24:26 2019] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 229 total children
[Wed Dec 04 14:24:27 2019] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 261 total children
[Wed Dec 04 14:24:28 2019] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 293 total children
[Wed Dec 04 14:24:40 2019] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 324 total children

--

Thanks & Regards,

Jayaram Ponnusamy 

ServerRoot "/apps/apache2.4"
Listen 80
<IfModule unixd_module>
	User daemon
	Group daemon
</IfModule>
ServerName stg.test
DocumentRoot "/apps/apache2.4/htdocs"
<Directory />
        Options FollowSymLinks
        AllowOverride All
        Require all granted
</Directory>
<Directory "/apps/apache2.4/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>
<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

ErrorLog "logs/error_log"
LogLevel info

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/opt/apps/apache/cgi-bin/"
</IfModule>

<Directory "/opt/apps/apache/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

Include conf/stg/stg.conf

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx