2016-05-24 21:47 GMT+03:00 Amos Jeffries <squid3@xxxxxxxxxxxxx>:
> On 25/05/2016 5:50 a.m., Deniz Eren wrote:
>> Hi,
>>
>> After upgrading to squid 3.5.16 I realized that squid started using
>> much of kernel's TCP memory.
>
> Upgrade from which version?
>
Upgrading from squid 3.1.14. I started using c-icap and ssl-bump.
>>
>> When squid was running for a long time TCP memory usage is like below:
>> test@test:~$ cat /proc/net/sockstat
>> sockets: used *
>> TCP: inuse * orphan * tw * alloc * mem 200000
>> UDP: inuse * mem *
>> UDPLITE: inuse *
>> RAW: inuse *
>> FRAG: inuse * memory *
>>
>> When I restart squid the memory usage drops dramatically:
>
> Of course it does. By restarting you just erased all of the operational
> state for an unknown but large number of active network connections.
>
That's true but what I mean was squid's CLOSE_WAIT connections are
using too much memory and they are not timing out.
> Whether many of those should have been still active or not is a
> different question. the answer to which depends on how you have your
> Squid configured, and what the traffic through it has been doing.
>
>
>> test@test:~$ cat /proc/net/sockstat
>> sockets: used *
>> TCP: inuse * orphan * tw * alloc * mem 10
>> UDP: inuse * mem *
>> UDPLITE: inuse *
>> RAW: inuse *
>> FRAG: inuse * memory *
>>
>
> The numbers you replaced with "*" are rather important for context.
>
>
Today again I saw the problem:
test@test:~$ cat /proc/net/sockstat
sockets: used 1304
TCP: inuse 876 orphan 81 tw 17 alloc 906 mem 29726
UDP: inuse 17 mem 8
UDPLITE: inuse 0
RAW: inuse 1
FRAG: inuse 0 memory 0
>> I'm using Squid 3.5.16.
>>
>
> Please upgrade to 3.5.19. Some important issues have been resolved. Some
> of them may be related to your TCP memory problem.
>
>
I have upgraded now and problem still exists.
>> When I look with "netstat" and "ss" I see lots of CLOSE_WAIT
>> connections from squid to ICAP or from squid to upstream server.
>>
>> Do you have any idea about this problem?
>
> Memory use by the TCP system of your kernel has very little to do with
> Squid. Number of sockets in CLOSE_WAIT does have some relation to Squid
> or at least to how the traffic going through it is handled.
>
> If you have disabled persistent connections in squid.conf then lots of
> closed sockets and FD are to be expected.
>
> If you have persistent connections enabled, then fewer closures should
> happen. But some will so expectations depends on how high the traffic
> load is.
>
Persistent connection parameters are enabled in my conf, the problem
occurs especially with connections to c-icap service.
My netstat output is like this:
netstat -tulnap|grep squid|grep CLOSE
tcp 211742 0 127.0.0.1:55751 127.0.0.1:1344
CLOSE_WAIT 17076/(squid-1)
tcp 215700 0 127.0.0.1:55679 127.0.0.1:1344
CLOSE_WAIT 17076/(squid-1)
tcp 215704 0 127.0.0.1:55683 127.0.0.1:1344
CLOSE_WAIT 17076/(squid-1)
...(hundreds)
Above ones are connections to c-icap service.
netstat -tulnap|grep squid|grep CLOSE
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 1 0 192.168.2.1:8443 192.168.6.180:45182
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.2.177:50020
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.2.172:60028
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.6.180:44049
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.6.180:55054
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.2.137:52177
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.6.180:43542
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.6.155:39489
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.0.147:38939
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.6.180:38754
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.0.164:39602
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.0.147:54114
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.6.180:57857
CLOSE_WAIT 15245/(squid-1)
tcp 1 0 192.168.2.1:8443 192.168.0.156:43482
CLOSE_WAIT 15245/(squid-1)
...(about 50)
Above ones are connections from https port to client.
As you can see recv-q for icap connections allocate more memory but
connections from https_port to upstream server connections allocate
only one byte.
What can be done to close these unused connections?
The problem in this thread seems similar:
http://www.squid-cache.org/mail-archive/squid-users/201301/0092.html
> Amos
>
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
