Hi,
Apache 2.2
Tomcat 7
Windows Server 2008 64 bit
I have a problem that has been dogging me for weeks and seems to be getting worse. I have Windows server and am fronting a Tomcat app server with Apache (using AJP connector) and SSL. This configuration has been in place for years without any issues.
Suddenly, somewhat randomly, during the day, users are experiencing long waits for server response. The difficult part is that it's difficult to reproduce. They'll say it's slow, I'll jump on, and everything is zippy fast for me. However, I have been able to experience it a few times. When I've seen it, I've seen the browser hang for a while (~10+ seconds) on "establishing secure connection". For one small period of time, the wait was so long, it was timing out on the client. When this was happening, I jumped onto the server and hit the Tomcat app locally (directly to Tomcat, bypassing Apache) and everything worked perfectly.. super fast. This makes me think it's something on the Apache side or at least the AJP connector.
Also note that during all of these episodes, no hardware seems to be taxed. Disk I/O, memory, CPU, etc all appear to be underutilized.
The only changes that have been made in recent months was that I have added a few new apps to Tomcat. This has not really increased users though. But I did have to added the additional ajp/proxy lines to the httpd.conf file and modify the rewrite rules in the virtual host config.
In trying to troubleshoot this problem, I also upped the MaxClients which didn't seem to have effect on anything.
Although it does *appear* to be load related. I could be wrong on that, but it seems most complaint tend be during the busier times of the day.
My hunch is that this is SSL related, but I really don't know. I've included the results from the server-status below... this was from one of those moments that it hung on "establishing secure connection" for a bit. I don't see anything obvious in it... but maybe someone here will.
Any ideas?
Thanks!!!
Apache Server Status for my.domain
Server Version: Apache/2.2.9 (Win32) mod_ssl/2.2.9 OpenSSL/0.9.8h
Server Built: Jun 13 2008 04:04:59
Current Time: Friday, 11-Apr-2014 11:36:32 Mountain Daylight Time
Restart Time: Friday, 11-Apr-2014 01:23:20 Mountain Daylight Time
Parent Server Generation: 0
Server uptime: 10 hours 13 minutes 12 seconds
Total accesses: 81852 - Total Traffic: 784.1 MB
2.22 requests/sec - 21.8 kB/second - 9.8 kB/request
64 requests currently being processed, 0 idle workers
KKKRKRRRRWCCRRRRRKRRKKKRRRRKRRRRRKRRKKRRKKRRKWKRKKCKKRRRKKKRKRKR
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M SS Req Conn Child Slot Client VHost Request
0-0 8184 1/744/744 K 6 8 0.0 5.98 5.98 the.ip.address www.my.domain GET /zzz2/polling?key=43DADEF1%2DB469%2DF802%2DD0C4%2D51DC8F2ED
0-0 8184 1/750/750 K 4 11 0.5 6.93 6.93 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 1/739/739 K 4 6 0.0 5.37 5.37 the.ip.address www.my.domain GET /zzz2/polling?key=4A8F4979%2DA41C%2D4C89%2DDFB3%2D51DFA7A4D
0-0 8184 0/822/822 R 28 2 0.0 7.39 7.39 ? ? ..reading..
0-0 8184 4/782/782 K 2 3 42.7 8.27 8.27 the.ip.address www.my.domain GET /zzz2/login.jsp HTTP/1.1
0-0 8184 0/790/790 R 27 12 0.0 9.60 9.60 ? ? ..reading..
0-0 8184 0/821/821 R 3 7 0.0 8.94 8.94 ? ? ..reading..
0-0 8184 0/749/749 R 19 6 0.0 4.54 4.54 ? ? ..reading..
0-0 8184 0/771/771 R 28 17 0.0 5.17 5.17 ? ? ..reading..
0-0 8184 0/1880/1880 W 0 0 0.0 15.27 15.27 the.ip.address www.my.domain GET /server-status HTTP/1.1
0-0 8184 2/814/814 C 0 10 0.6 5.57 5.57 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 1/853/853 C 0 50 2.1 7.29 7.29 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/824/824 R 19 10 0.0 5.96 5.96 ? ? ..reading..
0-0 8184 0/999/999 R 37 2 0.0 12.05 12.05 ? ? ..reading..
0-0 8184 0/796/796 R 25 7 0.0 6.91 6.91 ? ? ..reading..
0-0 8184 0/913/913 R 5 15 0.0 6.36 6.36 ? ? ..reading..
0-0 8184 0/839/839 R 1 998 0.0 10.81 10.81 ? ? ..reading..
0-0 8184 2/1550/1550 K 7 10 0.6 13.84 13.84 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/784/784 R 1 100 0.0 5.03 5.03 ? ? ..reading..
0-0 8184 0/1044/1044 R 15 161 0.0 11.32 11.32 ? ? ..reading..
0-0 8184 1/755/755 K 4 10 0.5 5.51 5.51 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 1/1875/1875 K 10 11 0.5 14.51 14.51 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 1/879/879 K 13 14 0.5 10.57 10.57 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/1818/1818 R 54 3 0.0 17.42 17.42 ? ? ..reading..
0-0 8184 0/823/823 R 53 4 0.0 6.85 6.85 ? ? ..reading..
0-0 8184 0/1637/1637 R 22 13 0.0 18.22 18.22 ? ? ..reading..
0-0 8184 0/775/775 R 39 12 0.0 5.04 5.04 ? ? ..reading..
0-0 8184 6/1485/1485 K 2 67 270.2 12.87 12.87 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/849/849 R 29 10 0.0 8.18 8.18 ? ? ..reading..
0-0 8184 0/1559/1559 R 23 13 0.0 17.91 17.91 ? ? ..reading..
0-0 8184 0/861/861 R 35 15 0.0 5.81 5.81 ? ? ..reading..
0-0 8184 0/1518/1518 R 19 10 0.0 19.21 19.21 ? ? ..reading..
0-0 8184 0/799/799 R 53 588 0.0 8.50 8.50 ? ? ..reading..
0-0 8184 2/1670/1670 K 1 11 0.6 18.14 18.14 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/868/868 R 34 11 0.0 6.91 6.91 ? ? ..reading..
0-0 8184 0/1678/1678 R 63 10 0.0 16.54 16.54 ? ? ..reading..
0-0 8184 1/843/843 K 9 8 0.0 5.37 5.37 the.ip.address www.my.domain GET /zzz2/polling?key=56615DD4%2D6BCD%2DABA7%2DAD3E%2D51DC6F70D
0-0 8184 2/1820/1820 K 12 13 0.6 26.00 26.00 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/998/998 R 52 12 0.0 10.28 10.28 ? ? ..reading..
0-0 8184 0/1692/1692 R 49 10 0.0 14.42 14.42 ? ? ..reading..
0-0 8184 2/961/961 K 6 8 0.6 12.39 12.39 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 3/1424/1424 K 9 50 122.7 13.46 13.46 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/893/893 R 21 12 0.0 10.42 10.42 ? ? ..reading..
0-0 8184 0/1674/1674 R 32 13 0.0 14.51 14.51 ? ? ..reading..
0-0 8184 1/1206/1206 K 9 20 0.5 12.10 12.10 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/1755/1755 W 3 0 0.0 15.76 15.76 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 1/945/945 K 9 12 0.5 7.12 7.12 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/1848/1848 R 16 8 0.0 18.11 18.11 ? ? ..reading..
0-0 8184 1/1835/1835 K 3 12 0.5 19.69 19.69 the.ip.address www.my.domain POST /scm/graniteamf/amf HTTP/1.1
0-0 8184 4/1584/1584 K 11 16 25.9 17.15 17.15 the.ip.address www.my.domain POST /scm/graniteamf/amf HTTP/1.1
0-0 8184 1/2037/2037 C 0 19 0.5 21.09 21.09 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 1/1803/1803 K 0 17 0.0 16.70 16.70 the.ip.address www.my.domain GET /zzz2/polling?key=7B3F544B%2D681F%2DCC7F%2D2B2C%2D51DCADC10
0-0 8184 1/1858/1858 K 4 10 0.0 18.39 18.39 the.ip.address www.my.domain GET /zzz2/polling?key=1C80514D%2D3B90%2DB3AD%2D3576%2D51DC80955
0-0 8184 0/1787/1787 R 14 8 0.0 19.95 19.95 ? ? ..reading..
0-0 8184 0/1512/1512 R 54 3 0.0 17.14 17.14 ? ? ..reading..
0-0 8184 0/1547/1547 R 42 11 0.0 14.27 14.27 ? ? ..reading..
0-0 8184 1/1727/1727 K 10 8 0.0 16.71 16.71 the.ip.address www.my.domain GET /zzz2/polling?key=BF88836B%2D1C9F%2DB024%2DFAF7%2D51DC6DF01
0-0 8184 1/1508/1508 K 6 11 0.5 12.32 12.32 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 1/1694/1694 K 13 10 0.5 17.53 17.53 the.ip.address www.my.domain POST /zzz2/graniteamf/amf HTTP/1.1
0-0 8184 0/1898/1898 R 18 0 0.0 18.54 18.54 ? ? ..reading..
0-0 8184 4/1961/1961 K 14 3 42.7 19.17 19.17 the.ip.address www.my.domain GET /zzz2/login.jsp HTTP/1.1
0-0 8184 0/1485/1485 R 46 3 0.0 12.38 12.38 ? ? ..reading..
0-0 8184 1/1979/1979 K 11 11 0.0 17.53 17.53 the.ip.address www.my.domain GET /zzz2/polling?key=CA3D01AB%2DD09A%2D3136%2D9A32%2D51DC5E83A
0-0 8184 0/1465/1465 R 48 10 0.0 10.84 10.84 ? ? ..reading..
Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M Mode of operation
SS Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn Kilobytes transferred this connection
Child Megabytes transferred this child
Slot Total megabytes transferred this slot
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 136
sub-caches: 32, indexes per sub-cache: 133
time left on oldest entries' SSL sessions: avg: 56 seconds, (range: 5...156)
index usage: 3%, cache usage: 4%
total sessions stored since starting: 9087
total sessions expired since starting: 8951
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 38783 hit, 12867 miss
total removes since starting: 0 hit, 0 miss
Apache/2.2.9 (Win32) mod_ssl/2.2.9 OpenSSL/0.9.8h Server at my.domain Port 443
