Hi. Sorry that I'm answering to the old thread. I was on vacation and didn't have a chance to test the proposed solution.
Dieter, yes, I'm on the old CentOS 6 based OS (Amazon Linux) but with a new kernel 4.9.27.
Amos, thank you for the suggestions about configure flags and squid config options, I fixed all issues you pointed to.
Unfortunately following workarounds didn't help:
* client_idle_pconn_timeout 30 seconds
* half_closed_clients on
* client_persistent_connections off
* server_persistent_connections off
However I assumed that this is a bug and that I can find older version which worked fine. I started testing from 3.1.x all the way to 3.5.26 and this is what I found:
* All versions until 3.5.21 work fine. There no issues with huge amount of TIME_WAIT connections under load.
* 3.5.20 is the latest stable version.
* 3.5.21 is the first broken version.
* 3.5.23, 3.5.25, 3.5.26 are broken as well.
This effectively means that bug is somewhere in between 3.5.20 and 3.5.21.
I hope this helps and I hope you'll be able to find an issue. If you can create a bug report based on this information and post it here it would be awesome.
Thank you.
On Wed, Jun 7, 2017 at 4:34 AM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote:
On 07/06/17 12:13, Ivan Larionov wrote:
Hi!
We recently updated from squid v2 to v3 and now see huge increase in connections in TIME_WAIT state on our squid servers (verified that this is clients connections).
The biggest change between 2.7 and 3.5 in this area is that 2.7 was HTTP/1.0 which closed TCP connections after each request by default, and 3.5 is HTTP/1.1 which does not. So connections are more likely to persist until they hit some TCP timeout then enter the slow TIME_WAIT process.
There were also some other bugs identified in older 3.5 releases which increased the TIME_WAIT specifically. I thought those were almost all fixed by now, but YMMV whether you hit the remaining issues.
A workaround it to set <http://www.squid-cache.org/Doc/config/client_idle_pconn_tim > to a shorter value than the default 2min. eg you might want it to be 30sec or so.eout/
See versions and amount of such connections under the same load with the same configs (except some incompatible stuff):
squid 2.7.STABLE9
configure options: '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--bindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--datadir=/usr/share' '--sysconfdir=/etc/squid' '--enable-epoll' '--enable-removal-policies=heap,lru' '--enable-storeio=aufs' '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests' '--enable-useragent-log' '--enable-referer-log' '--with-large-files' '--with-maxfd=16384' '--enable-err-languages=Englis h'
# netstat -tn | grep TIME_WAIT | grep 3128 | wc -l
95
squid 3.5.25
configure options: '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--sysconfdir=/etc/squid' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid' '--includedir=/usr/include' '--datadir=/usr/share' '--sharedstatedir=/usr/com' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-epoll' '--enable-removal-policies=heap,lru' '--enable-storeio=aufs' '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests' '--enable-useragent-log' '--enable-referer-log' '--with-large-files' '--with-maxfd=16384' '--enable-err-languages=Englis h' '--enable-htcp'
FYI, these options are not doing anything for Squid-3:
'--enable-useragent-log' '--enable-referer-log' '--enable-err-languages=English'
# netstat -tn | grep TIME_WAIT | grep 3128 | wc -l
11277
Config:
http_port 0.0.0.0:3128 <http://0.0.0.0:3128>
acl localnet src 10.0.0.0/8 <http://10.0.0.0/8> # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 <http://172.16.0.0/12> # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 <http://192.168.0.0/16> # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 1025-65535 # unregistered ports
acl CONNECT method CONNECT
### START CUSTOM
acl Purge_method method PURGE
# Allow localhost to selectively flush the cache
http_access allow localhost Purge_method
http_access deny Purge_method
### END CUSTOM
### ALLOW ACCESS TO ALL PORTS
# http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
### START CUSTOM
# Disable icp
icp_port 0
# Allow ICP queries from local networks only
icp_access allow localnet
icp_access allow localhost
icp_access deny all
# Disable htcp
htcp_port 0
# Allow HTCP queries from local networks only
htcp_access allow localnet
htcp_access allow localhost
htcp_access deny all
FYI: setting icp_access and htcp_access is pointless when the relevant port is 0. That port 0 disables the entire component.
# Check for custom request header
acl custom_acl req_header x-use-custom-proxy -i true
# Check for x-use-new-proxy request header
acl custom_new_acl req_header x-use-new-proxy -i true
# first_proxy
cache_peer 127.0.0.1 parent 18070 0 no-query no-digest name=first_proxy
cache_peer_access first_proxy deny custom_acl
cache_peer_access first_proxy deny custom_new_acl
# second_proxy
cache_peer 127.0.0.1 parent 18079 0 no-query no-digest name=second_proxy
cache_peer_access second_proxy allow custom_acl
cache_peer_access second_proxy allow custom_new_acl
cache_peer_access second_proxy deny all
never_direct allow all
cache_mem 4620591 KB
maximum_object_size_in_memory 8 KB
memory_replacement_policy heap LRU
cache_replacement_policy heap LRU
cache_dir aufs /mnt/services/squid/cache 891289 16 256
minimum_object_size 64 bytes # none-zero so we dont cache mistakes
maximum_object_size 102400 KB
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %tr "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
Please do not re-define these formats. If you want to use the default format they are defined internally by Squid3, if you want any customizations use a different format name.
access_log stdio:/var/log/squid/access.log combined
cache_log /var/log/squid/cache.log
cache_store_log none
logfile_rotate 0
client_db off
pid_filename /var/run/squid.pid
coredump_dir /var/cache
### END CUSTOM
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
# refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
Please do not remove that cgi-bin pattern. It is there to protect the cache against servers with broken/ancient CGI engines. It is designed explicitly so modern dynamic sites that provide proper cacheability headers can still be stored. So no harm and only benefits from in leaving it there.
Amos
_______________________________________________
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
With best regards, Ivan Larionov.
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
