Dear All,
We were runnig our site in PHP based CMS tool earlier, and normally 20-30K users will access our sites daily. But in new system with Tomcat, we are facing performance and availability issue frequently, when i access the tomcat url directly the page is loading within 3seconds, but if we access webServer URL then its taking more than 9seconds.
Also, Each day I am seeing more and more of these in my error_logs, and when the Total Children value is reached 999 the Apache is not responding and Server reboot only help to bring the site back. Every day atleast 4-5 times we are facing this issue (we are using mod_jk to connect with tomcat).
Kindly please help on this.
Usually I am seeing this on my error_log:
[Sat Apr 15 20:49:33 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 4 idle, and 31 total children
[Sat Apr 15 20:51:14 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 20 total children
[Sat Apr 15 20:51:15 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 0 idle, and 28 total children
[Sat Apr 15 20:51:16 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 44 total children
We are using two Apache Nodes and Connected with Two Tomcat (at Application Level Clustering).
Apache Servers:
4 Core 64-bit, Rhel System running on 16GB RAM (Both Servers)
Server version: Apache/2.2.21 (Unix)
httpd.conf
KeepAlive On
Timeout 300
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule prefork.c>
StartServers 80
ServerLimit 3500
MaxClients 3500
MaxRequestsPerChild 0
</IfModule>
workers.properties
worker.list=loadbalancer,status
worker.qu_prod_live_svr.type=ajp13
worker.qu_prod_live_svr.host=cmsp1
worker.qu_prod_live_svr.port=8009
worker.qu_prod_live_svr.socket_keepalive=1
worker.qu_prod_live_svr.socket_timeout=300
worker.qu_prod_live_svr1.type=ajp13
worker.qu_prod_live_svr1.host=cmsp2
worker.qu_prod_live_svr1.port=8009
worker.qu_prod_live_svr1.socket_keepalive=1
worker.qu_prod_live_svr1.socket_timeout=300
worker.qu_prod_live_svr.lbfactor=1
worker.qu_prod_live_svr1.lbfactor=1
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=qu_prod_live_svr,qu_prod_live_svr1
worker.status.type=status
Tomcat Servers:
4 Core 64-bit, Rhel System running on 16GB RAM (Both Servers)
Server version: Apache Tomcat/7.0.42
<Connector port="9090" protocol="HTTP/1.1" redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" maxThreads="500" minSpareThreads="10" connectionTimeout="-1" />
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" />
error_log:
[Sat Apr 15 21:52:36 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 839 total children
[Sat Apr 15 21:52:37 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 871 total children
[Sat Apr 15 21:52:38 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 903 total children
[Sat Apr 15 21:52:39 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 935 total children
[Sat Apr 15 21:52:40 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 967 total children
[Sat Apr 15 21:52:41 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 999 total children
[Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Sat Apr 15 21:52:41 2017] [alert] Child 9351 returned a Fatal error... Apache is exiting!
[Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_lock(jk_log_lock) failed
[Sat Apr 15 21:53:06 2017] [error] mod_jk: jk_log_to_file
[Sat Apr 15 21:53:06 2017][8752:4177577728] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1150): (qu_prod_live_svr1) can't receive the response header message from tomcat, network problems or tomcat (
10.11.11.32:8009) is down (errno=104)\n failed: Broken pipe
[Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_unlock(jk_log_lock) failed
[Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_lock(jk_log_lock) failed
[Sat Apr 15 21:53:06 2017] [error] mod_jk: jk_log_to_file [Sat Apr 15 21:53:06 2017][8752:4177577728] [error] ajp_get_reply::jk_ajp_common.c (1962): (qu_prod_live_svr1) Tomcat is down or refused connection. No response has been sent to the client (yet)\n failed: Broken pipe
[Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_unlock(jk_log_lock) failed
