Also, I just found the following in my dmesg output:
httpd[8313]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10526]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10532]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10530]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10533]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10543]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10546]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10549]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10550]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10557]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10556]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9678]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10554]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9483]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[10527]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9450]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[4952]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[8736]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[8348]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[10561]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[7064]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[8752]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9803]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[7882]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9930]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9988]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10018]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[10040]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[10035]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[9874]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[6971]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9400]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[10574]: segfault at 00007fff62fa8ff8 rip 0000000000441479 rsp 00007fff62fa8fe0 error 6
httpd[9914]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
httpd[9067]: segfault at 00007fffbbe1dfe8 rip 0000000000441459 rsp 00007fffbbe1dfb0 error 6
I have successfully and consistently run 2.25 million servlet transactions out of 45 million+ requests through a hardware balancer across 8 clusters of 2 jvms for a total of 16 tomcat instances running on two different server grade machines in ~1 hour with little or no errors. Each tomcat instance uses ~160 busy threads and most transactions were under 2 seconds. The boxes ran at about 50% utilized.
When I try to run the same test against the same setup but with two apache 2.2.4 instances running on a third box, I can successfully proxy to and balance across the clusters, but once the volume ramps-up I run into many issues. Now, tomcat easily uses more than 1000 busy threads.
I get different issues with mod_proxy_ajp and mod_proxy_http.
With mod_proxy_ajp, ~5 to 10 minutes into the test mod_proxy_ajp transaction times go through the roof and the apache error log complains about:
[Wed May 23 05:44:43 2007] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed May 23 05:48:20 2007] [error] (70014)End of file found: ajp_ilink_receive() can't receive header
[Wed May 23 05:48:20 2007] [error] ajp_read_header: ajp_ilink_receive failed
[Wed May 23 05:48:20 2007] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (xxx.xxx.xxx.xxx)
[Wed May 23 05:48:25 2007] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
With mod_proxy_http everything seems to run fine until about 30 minutes into the test the apache error logs complain about:
[Wed May 23 02:41:07 2007] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed May 23 03:18:41 2007] [error] [client xxx.xxx.xxx.xxx] proxy: error reading status line from remote server xxx.xxx.xxx.xxx, referer: http://xxx.xxx.xxx.xxx.....
[Wed May 23 03:18:41 2007] [error] [client xxx.xxx.xxx.xxx] proxy: Error reading from remote server returned by /tf/tad/accountSearch, referer: http://xxx.xxx.xxx.xxx....
[Wed May 23 03:18:42 2007] [error] [client xxx.xxx.xxx.xxx] proxy: error reading status line from remote server xxx.xxx.xxx.xxx, referer: http://xxx.xxx.xxx.xxx....
[Wed May 23 03:18:42 2007] [error] [client xxx.xxx.xxx.xxx] proxy: Error reading from remote server returned by /tf/tad/loadPageFragment, referer: http://xxx.xxx.xxx.xxx....
[Wed May 23 03:18:42 2007] [error] [client xxx.xxx.xxx.xxx] proxy: error reading status line from remote server xxx.xxx.xxx.xxx, referer: http://xxx.xxx.xxx.xxx.....
[Wed May 23 03:18:42 2007] [error] [client xxx.xxx.xxx.xxx] proxy: Error reading from remote server returned by /tf/tad/accountSearch, referer: http://xxx.xxx.xxx.xxx....
[Wed May 23 03:18:42 2007] [error] [client xxx.xxx.xxx.xxx] (70007)The timeout specified has expired: proxy: error reading response
If I let the mod_proxy_http test run it eventually brings down the two application servers to the point where I have to reboot them.
Here are my apache settings for mod_proxy_http (mod_proxy_ajp is the same except it uses the ajp protocol):
<Proxy balancer://mybalancer>
BalancerMember http://1.2.3.4:8201/tf/tad smax=400 max=500 route=b1 redirect=a1
BalancerMember http://1.2.3.5:8201/tf/tad smax=400 max=500 route=a1 redirect=b1
BalancerMember http://1.2.3.4:8202/tf/tad smax=400 max=500 route=b2 redirect=a2
BalancerMember http://1.2.3.5:8202/tf/tad smax=400 max=500 route=a2 redirect=b2
BalancerMember http://1.2.3.4:8203/tf/tad smax=400 max=500 route=b3 redirect=a3
BalancerMember http://1.2.3.5:8203/tf/tad smax=400 max=500 route=a3 redirect=b3
BalancerMember http://1.2.3.4:8204/tf/tad smax=400 max=500 route=b4 redirect=a4
BalancerMember http://1.2.3.5:8204/tf/tad smax=400 max=500 route=a4 redirect=b4
BalancerMember http://1.2.3.4:8205/tf/tad smax=400 max=500 route=b5 redirect=a5
BalancerMember http://1.2.3.5:8205/tf/tad smax=400 max=500 route=a5 redirect=b5
BalancerMember http://1.2.3.4:8206/tf/tad smax=400 max=500 route=b6 redirect=a6
BalancerMember http://1.2.3.5:8206/tf/tad smax=400 max=500 route=a6 redirect=b6
BalancerMember http://1.2.3.4:8207/tf/tad smax=400 max=500 route=b7 redirect=a7
BalancerMember http://1.2.3.5:8207/tf/tad smax=400 max=500 route=a7 redirect=b7
BalancerMember http://1.2.3.4:8208/tf/tad smax=400 max=500 route=b8 redirect=a8
BalancerMember http://1.2.3.5:8208/tf/tad smax=400 max=500 route=a8 redirect=b8
</Proxy>
ProxyPass /my/servlet balancer://mybalancer stickysession=JSESSIONID nofailover=Off maxattempts=2 lbmethod=byrequests
<IfModule mpm_prefork_module>
StartServers 512
ServerLimit 1024
MaxClients 1024
MinSpareServers 512
MaxSpareServers 512
MaxRequestsPerChild 512
</IfModule>
Server version: Apache/2.2.4 (Unix)
Server built: May 14 2007 12:26:06
Compiled in modules:
core.c
mod_authn_file.c
mod_authn_default.c
mod_authz_host.c
mod_authz_groupfile.c
mod_authz_user.c
mod_authz_default.c
mod_auth_basic.c
mod_include.c
mod_filter.c
mod_log_config.c
mod_env.c
mod_setenvif.c
mod_proxy.c
mod_proxy_connect.c
mod_proxy_ftp.c
mod_proxy_http.c
mod_proxy_ajp.c
mod_proxy_balancer.c
prefork.c
http_core.c
mod_mime.c
mod_status.c
mod_autoindex.c
mod_asis.c
mod_cgi.c
mod_negotiation.c
mod_dir.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_so.c
Hardware/OS setup for all 3 boxes:
Number of CPUs per Server: 4 Dual-core AMD Opteron
Memory per Server: 16 gigabyte
OS: Suse Enterprise Linux (SLES) Version 10.2
Is this a bug or am I configuring this wrong? Any configuration suggestions would be appreciated
Thanks,
J
____________________________________________________________________________________Got a little couch potato?
Check out fun summer activities for kids.
http://search.yahoo.com/search?fr=oni_on_mail&p=summer+activities+for+kids&cs=bz
____________________________________________________________________________________Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
http://farechase.yahoo.com/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|