On 27.09.2012 07:09, tcr@xxxxxxxxxxxx wrote:
Hi Eliezer,
Thanks for the feedback. This memory leak causes real-world problems
for me, as squid starts to do a lot of swapping when it exhausts
physical RAM, and things slow down. Also, the sheer magnitude of the
memory numbers is just ridiculous... squid easily grows to in excess
of 10GB of resident memory as shown in top. I've got the servers
restarting their squids periodically, but that's a pretty bad
solution.
Here is my squid.conf. Note two include files...
squid_ns5_allowed_ips.conf and squid_blacklist_ips.conf . These are
lists of IPs in an ACL. The allowed IPs one has lots of entries
(almost 20,000) and that's the only thing I think is really unusual
about my setup, so I'm wondering if that is exposing a leak
somehwere.
##### BEGIN squid.conf #######
http_port 5000
http_port 5001
http_port 5002
http_port 5003
http_port 5020
http_port 5021
http_port 5022
http_port 5023
cache_mgr [omitted]
visible_hostname [omitted]
No need to omit hostname. This is the *public* FQDN which your squid
uses in error pages to load icon and sub-object URLs.
max_filedesc 32768
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
Dropping "QUERY" is a good idea. Most of the web is now dynamic pages
which this will block and a lot of that actually provides caching
information Squid-2.7+ can use. I've found the 10-20% HIT barrier to be
mostly caused by these rules.
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
Add:
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
NP: this replaces the "QUERY" ACLs and makes squid act properly when
handed dynamic content without proper caching controls.
refresh_pattern . 0 20% 4320
NP: the following refresh_pattern are duplicates, and would never be
used due to the '.' pattern above.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
acl demo_sites dstdomain .raynersoftware.com
acl banned_sites dstdomain client.versiontracker.com
acl banned_sites dstdomain .rapidshare.com
# allow yuki2
acl a_ips src 64.62.244.50/32
include /etc/squid/squid_ns5_allowed_ips.conf
include /etc/squid/squid_blacklist_ips.conf
# acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
NP: when you move to 3.2 the above are also built-in ACLs.
acl SSL_ports port 443 563
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 443 563
acl Safe_ports port 70
acl Safe_ports port 210
acl Safe_ports port 1025-65535
acl Safe_ports port 280
acl Safe_ports port 488
acl Safe_ports port 591
acl Safe_ports port 777
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
http_access deny banned_sites
http_access deny banned_ips
http_access allow demo_sites
# we shorten allowed_ips to a_ips to save space in our include file
http_access allow a_ips
Optimization:
If you pre-filter your a_ips details to remove banned_ips, you shoudl
be able to convert to:
http_access allow demo_sites !banned_ips
http_access allow a_ips
http_access deny all
http_reply_access allow all
#icp_access allow all
pid_filename /var/run/squid.pid
coredump_dir /var/spool/squid
via off
forwarded_for off
##### END squid.conf #######
When I do a mgr:info on one particular box, I get this:
Memory usage for squid via mallinfo():
Total space in arena: 1101288 KB
Ordinary blocks: 1098718 KB 287 blks
Small blocks: 0 KB 0 blks
Holding blocks: 17636 KB 9 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 2569 KB
Total in use: 1116354 KB 100%
Total free: 2569 KB 0%
Total size: 1118924 KB
Memory accounted for:
Total accounted: 284679 KB 25%
memPool accounted: 284679 KB 25%
memPool unaccounted: 834245 KB 75%
memPoolAlloc calls: 620095300
memPoolFree calls: 640694955
Yet in ps, I see this:
squid 2576 1.1 32.5 5388508 5314184 ? S Sep05 92:51
(squid) -f /etc/squid/squid.conf
So, squid is using 5GB of RAM yet it only knows about 1 of those GB.
FYI: mallinfo() function used by Squid to display some of those memory
statistics is known to contain a 32-bit wrap problem on 64-bit OS.
Since you said your Squid is at 5GB memory usage that is 1x 4GB wraps
and the remainder (+1 GB) is what we can expect to see mallinfo()
reporting as total. It may just be that which you are seeing.
NP: "Memory accounted for" is the Squid internal accounting. 284MB is
on record as currently in-use, with "unaccounted" being the mallinfo()
value minus the Squid "total accounted" (when 32-bit wrap happens this
shows large negative numbers, which is a dead giveaway. You are in the
fuzzy area of +N remainders on 32-bit wrap where it is unclear).
To be absolutely certain of memory leaks, build with valgrind support.
The cache mgr memory reports get appended with valgrind 'snapshot'
reports of usage/leaks and IIRC something for global leaks reports is
logged on shutdown.
NP: its about time we ran 3.2 through another valgrind session, your
traffic looks high enough to provide useful details. If you are happy to
do this the valgrind report will be welcomed on the squid-dev mailing
list.
Amos
