On 29.11.2010 23:25, Travis Whitton wrote:
Hi,
We're experiencing some odd behavior regarding connections taking a
long time to establish to our website. We've been running Apache in
production for over three years now and have recently began
experiencing issues where the server-status page, static, and dynamic
content response times will slow anywhere from a few seconds to long
enough for the connection to timeout.
Initially thinking we might be hitting some hard limits with the OS,
we've thoroughly audited our sysctl variables, tried disabling
iptables and conntrack, and ensured that we're not running out of
ephemeral ports or anything along those lines. Looking at netstat, it
seems we have a pretty large number of connections in TIME_WAIT which
is understandable since this is a high traffic website, but I'm
wondering if this value could indicate we're backlogging on TCP
connections or something along those lines?
[root@RHL073 ipv4]# netstat -an | awk '/^tcp/ {A[$(NF)]++} END {for (I
in A) {printf "%5d %s\n", A[I], I}}'
34723 TIME_WAIT
3 CLOSE_WAIT
275 FIN_WAIT1
74 FIN_WAIT2
8824 ESTABLISHED
815 SYN_RECV
102 CLOSING
30 LAST_ACK
10 LISTEN
In an effort to tune things, I've tried playing with the TCP timeout
settings a bit, and the response times have improved somewhat. Please
note that I've been testing response times using the loopback
interface to rule out any ethernet hardware issues.
echo 15> /proc/sys/net/ipv4/tcp_fin_timeout
echo 1> /proc/sys/net/ipv4/tcp_tw_recycle
echo 1> /proc/sys/net/ipv4/tcp_tw_reuse
We're running prefork, and have configured the client settings to what
seem to be reasonable limits for our hardware.
<IfModule prefork.c>
StartServers 100
MinSpareServers 100
MaxSpareServers 200
ServerLimit 1500
MaxClients 1500
MaxRequestsPerChild 1000000
</IfModule>
Any help or advice would be greatly appreciated.
Yes, having lots of TME_WAIT can have a serious impact on TCP performance. So I think your approach is reasonable. Unfortunately (at leas that was true a few years ago), Linux does not support setting a timeout value for TIME_WAIT, as e.g. Solaris does. Unfortunately the docs about the reuse and recycle switches is far from being detailed.
Are you using HTTP Keep-Alive? Your high ESTABLISHED numbers suggest that. If not, that could reduce the TIME_WAIT numbers too, but comes with a price: you would get much higher ESTABLISHED rates (and thus need for even more httpd threads, typically about 5 times of what you see without Keep-Alive).
Regards, Rainer --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|