I started getting problem reports that clients were timing out when connecting -- from our client logs I could see that it sometimes was taking between five and 15 minutes to download a 200-byte file. (During normal times, of course, the file downloads in less than 1 second.) These long downloads happen on client machines on customer networks, AND on client machines on our internal network.
I've tweaked Apache configs to make the problem less severe, but my main problem is that I still have NO IDEA what's going on here -- what's causing the problem or how I can fix it permanently. (If it's a problem with a customer network/firewall, I would love to know what to tell them -- and since we can't count on having properly configured customer networks, we need to be able to minimize this problem going forward.)
The Apache processes appear to be getting stuck in "Read" mode, and are not released. At our heaviest, I'm seeing somewhere around 100-150 connections per MINUTE in access.log -- which should not be overwhelming Apache.
I'm looking for an answer as to what's going on here, or suggestions for further investigation and diagnosis.
The Apache server's internal ip is 192.168.61.10. It's on a DMZ; internal network is 192.168.60.x. Firewall/router is a Juniper Netscreen 25.
I hacked up a script to download a test file every five seconds, and log server status when it failed the download. This is that script:
#!/bin/bash ETERNITY="TRUE" while [ $ETERNITY = "TRUE" ] do echo echo "About to test stark at `date`."curl --connect-timeout 10 --max-time 25 -s -S --stderr /Users/ schof/Desktop/problems.log http://example.dakim.com/testfile > /dev/ nullif [ $? -ne "0" ]; then echo "Test FAILED at `date`"ssh root@xxxxxxxxxxxxxxxxx netstat -n -p TCP >> /Users/schof/ Desktop/NetstatOutput`date +%s`.txt curl http://example.dakim.com/server-status >> /Users/schof/ Desktop/ServerStatus`date +%s`.htmlelse echo "Test succeeded at `date`." sleep 5 fi done
Here's a section of the netstat output: (I've replaced customer IP addresses with obviously incorrect ones. (All 444.444.444.444 entries were from the same IP address.)
Active Internet connections (w/o servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.61.10:80 444.444.444.444:49576 SYN_RECV - tcp 0 0 192.168.61.10:80 444.444.444.444:49577 SYN_RECV - tcp 0 0 192.168.61.10:80 192.168.61.1:27486 SYN_RECV - tcp 0 0 192.168.61.10:80 555.555.555.555:49952 SYN_RECV - tcp 0 0 192.168.61.10:80 444.444.444.444:49497 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52825 ESTABLISHED22939/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49496 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52824 ESTABLISHED22968/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49499 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52827 ESTABLISHED22939/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49498 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52826 ESTABLISHED22997/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49501 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52829 ESTABLISHED22997/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49500 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52828 ESTABLISHED22968/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49503 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52831 ESTABLISHED22968/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49502 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52830 ESTABLISHED22939/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:52817 ESTABLISHED22968/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49489 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52816 ESTABLISHED22910/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49488 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52819 ESTABLISHED22910/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49491 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52818 ESTABLISHED22939/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49490 ESTABLISHED- tcp 0 0 192.168.61.10:80 444.444.444.444:52821 ESTABLISHED22939/apache2 tcp 0 0 192.168.61.10:80 444.444.444.444:49493 ESTABLISHED-
And here's a section of the Apache Extended Server-Status page:
Apache Server Status for example.dakim.com Server Version: Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.1.4 Server Built: Jul 28 2006 08:55:39 Current Time: Monday, 25-Jun-2007 20:21:27 PDT Restart Time: Sunday, 24-Jun-2007 06:01:46 PDT Parent Server Generation: 0 Server uptime: 1 day 14 hours 19 minutes 41 seconds Total accesses: 156162 - Total Traffic: 1.4 GB CPU Usage: u.03 s.02 cu0 cs0 - 3.62e-5% CPU load 1.13 requests/sec - 10.5 kB/second - 9.3 kB/request 150 requests currently being processed, 0 idle workers RRRRRRRRRRRRRRRRRRRRRRRRR....................................... RRRRRRRRRRRRRRRRRRRRRRRRR....................................... RRRRRRRRRRRRRRRRRRRRWRRRR....................................... RRRRRRRRRRRRRRRRRRRRRRRRR....................................... RRRRRRRRRRRRRRRRRRRRRRRRR....................................... RRRRRRRRRRRRRRRRRRRRRRRRR....................................... ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request 0-0 23058 0/0/3122 R 0.00 11 0 0.0 0.00 3.75 ? ? ..reading.. 0-0 23058 0/0/2853 R 0.00 11 0 0.0 0.00 0.16 ? ? ..reading.. 0-0 23058 0/0/2826 R 0.00 10 0 0.0 0.00 -1652.90 ? ? ..reading.. 0-0 23058 0/0/2841 R 0.00 10 0 0.0 0.00 92.31 ? ? ..reading.. 0-0 23058 0/0/2722 R 0.00 9 0 0.0 0.00 0.31 ? ? ..reading.. 0-0 23058 0/3/2777 R 0.00 16 0 0.0 0.00 0.32 ? ? ..reading.. 0-0 23058 0/0/2707 R 0.00 15 0 0.0 0.00 32.64 ? ? ..reading.. 0-0 23058 0/1/2731 R 0.00 15 0 0.0 0.00 0.11 ? ? ..reading.. 0-0 23058 0/0/2736 R 0.00 14 0 0.0 0.00 1.93 ? ? ..reading.. 0-0 23058 0/0/2760 R 0.00 9 0 0.0 0.00 0.31 ? ? ..reading.. 0-0 23058 0/0/2748 R 0.00 7 0 0.0 0.00 0.15 ? ? ..reading.. 0-0 23058 0/0/2733 R 0.00 6 0 0.0 0.00 0.11 ? ? ..reading.. 0-0 23058 0/0/2756 R 0.00 14 0 0.0 0.00 6.89 ? ? ..reading.. 0-0 23058 0/2/2810 R 0.00 13 0 0.0 0.00 12.00 ? ? ..reading.. 0-0 23058 0/0/2751 R 0.00 13 0 0.0 0.00 0.12 ? ? ..reading.. 0-0 23058 0/0/2770 R 0.00 6 0 0.0 0.00 46.61 ? ? ..reading.. 0-0 23058 0/0/2781 R 0.00 13 0 0.0 0.00 0.33 ? ? ..reading..
I can put the complete netstat and server-status files up for DL if anyone requests.
Thanks very much! --John Schofield --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|