Recently due to squid 2.7 being EOL'ed, we migrated our squid server to version 3.4.9 on a FreeBSD 10.0-RELEASE running on 64-bit hardware. We started seeing paging file being swapped out eventually running out of available memory. From the time squid gets started it usually takes about two days before we see these entries in /var/log/messages as follows: +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(6): failed +swap_pager_getswapspace(16): failed Looking at the 'top' results, I see that the swap file has been totally exhausted. Memory used by squid hovers around 2.3GB out of the total 3GB of system memory. I am not sure what is causing these memory leaks. After rebooting, squid-internal-mgr/info shows the following statistics: Squid Object Cache: Version 3.4.9 Build Info: Start Time: Mon, 24 Nov 2014 18:39:08 GMT Current Time: Mon, 24 Nov 2014 19:39:13 GMT Connection information for squid: Number of clients accessing cache: 18 Number of HTTP requests received: 10589 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 176.2 Average ICP messages per minute since start: 0.0 Select loop called: 763993 times, 4.719 ms avg Cache information for squid: Hits as % of all requests: 5min: 3.2%, 60min: 17.0% Hits as % of bytes sent: 5min: 2.0%, 60min: 6.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 37.2% Disk hits as % of hit requests: 5min: 22.2%, 60min: 33.2% Storage Swap size: 7361088 KB Storage Swap capacity: 58.5% used, 41.5% free Storage Mem size: 54348 KB Storage Mem capacity: 3.9% used, 96.1% free Mean Object Size: 23.63 KB Requests given to unlinkd: 1 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.10857 0.19742 Cache Misses: 0.10857 0.32154 Cache Hits: 0.08265 0.01387 Near Hits: 0.15048 0.12106 Not-Modified Replies: 0.00091 0.00091 DNS Lookups: 0.05078 0.05078 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 3605.384 seconds CPU Time: 42.671 seconds CPU Usage: 1.18% CPU Usage, 5 minute avg: 0.72% CPU Usage, 60 minute avg: 1.17% Maximum Resident Size: 845040 KB Page faults with physical i/o: 20 Memory accounted for: Total accounted: 105900 KB memPoolAlloc calls: 2673353 memPoolFree calls: 2676487 File descriptor usage for squid: Maximum number of file descriptors: 87516 Largest file desc currently in use: 310 Number of file desc currently in use: 198 Files queued for open: 0 Available number of file descriptors: 87318 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 311543 StoreEntries 4421 StoreEntries with MemObjects 4416 Hot Object Cache Items 311453 on-disk objects I will post another one tomorrow that will indicate growing memory/swapfile consumption. Here is my squid.conf: # OPTIONS FOR AUTHENTICATION # ----------------------------------------------------------------------------- # 1st four lines for auth_param basic children 5 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours auth_param basic casesensitive off # next three lines for kerberos authentication (needed to use usernames) # used in conjunction with "acl auth proxy_auth" line below #auth_param negotiate program /usr/local/libexec/squid/negotiate_kerberos_auth -i #auth_param negotiate children 50 startup=10 idle=5 #auth_param negotiate keep_alive on # ACCESS CONTROLS # ----------------------------------------------------------------------------- # Example rule allowing access from your local networks. # Adapt to list your (internal) IP networks from where browsing # should be allowed #acl manager proto cache_object acl manager url_regex -i ^cache_object:// /squid-internal-mgr/ acl adminhost src 192.168.1.149 acl localnet src 192.168.1.0/24 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl webserver src 198.168.1.35 acl some_big_clients src 192.168.1.149/32 #CI53 # We want to limit downloads of these type of files # Put this all in one line acl magic_words url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4 .img # We don't block .html, .gif, .jpg and similar files, because they # generally don't consume much bandwidth acl SSL_ports port 443 acl SSL_ports port 8443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT acl CONNECT method CONNECT acl Winupdate dstdomain .microsoft.com # Dot is important acl social-sites dstdomain .facebook.com .fbcdn.net acl ZipInfo dstdomain .dial-a-zip.com # in conjunction with negotiate_kerberos_auth line above #acl auth proxy_auth REQUIRED # # Recommended minimum Access Permission configuration: # http_access allow manager localnet http_access allow manager localhost http_access allow manager webserver http_access allow manager adminhost http_access deny manager acl PURGE method PURGE http_access allow PURGE localhost http_access deny PURGE # Deny requests to certain unsafe ports http_access deny !Safe_ports # Deny CONNECT to other than secure SSL ports http_access deny CONNECT !SSL_ports # We strongly recommend the following be uncommented to protect innocent # web applications running on the proxy server who think the only # one who can access services on "localhost" is a local user #http_access deny to_localhost # # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS # redirector_access deny Winupdate # in conjunction with negotiate_kerberos_auth line above #http_access deny !auth #http_access allow auth # Example rule allowing access from your local networks. # Adapt localnet in the ACL section to list your (internal) IP networks # from where browsing should be allowed http_access allow localnet http_access allow localhost http_access allow ZipInfo localnet # And finally deny all other access to this proxy http_access deny all # NETWORK OPTIONS # ----------------------------------------------------------------------------- # Squid normally listens to port 3128 http_port 3128 # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM # ----------------------------------------------------------------------------- hierarchy_stoplist cgi-bin ? # MEMORY CACHE OPTIONS # ----------------------------------------------------------------------------- cache_mem 1366 MB #cache_mem 2134 MB #maximum_object_size_in_memory 64 KB maximum_object_size_in_memory 128 KB # DISK CACHE OPTIONS # ----------------------------------------------------------------------------- cache_replacement_policy heap LFUDA cache_dir aufs /data/squid/aufs_cache 4096 16 256 min-size=131073 cache_dir diskd /data/squid/diskd_cache 8192 16 256 Q1=64 Q2=72 max-size=131072 #maximum_object_size 122880 KB maximum_object_size 153600 KB cache_swap_low 90 cache_swap_high 95 # LOGFILE OPTIONS # ----------------------------------------------------------------------------- access_log daemon:/data/squid/logs/access.log cache_store_log daemon:/data/squid/logs/store.log cache_swap_log /var/spool/squid/%s logfile_rotate 28 # OPTIONS FOR TROUBLESHOOTING # ----------------------------------------------------------------------------- cache_log /data/squid/logs/cache.log # Leave coredumps in the first cache dir coredump_dir /data/squid # OPTIONS FOR EXTERNAL SUPPORT PROGRAMS # ----------------------------------------------------------------------------- diskd_program /usr/local/libexec/squid/diskd # OPTIONS FOR TUNING THE CACHE # ----------------------------------------------------------------------------- refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160 refresh_pattern http://office\.microsoft\.com/ 0 80% 20160 refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160 refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160 refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160 refresh_pattern http://download\.microsoft\.com/ 0 80% 20160 refresh_pattern http://download\.macromedia\.com/ 0 80% 20160 refresh_pattern http://ftp\.software\.ibm\.com/ 0 80% 20160 refresh_pattern cgi-bin 1 20% 2 refresh_pattern \.asp$ 1 20% 2 refresh_pattern \.acgi$ 1 20% 2 refresh_pattern \.cgi$ 1 20% 2 refresh_pattern \.pl$ 1 20% 2 refresh_pattern \.shtml$ 1 20% 2 refresh_pattern \.php3$ 1 20% 2 refresh_pattern \? 1 20% 2 refresh_pattern \.gif$ 10080 90% 43200 refresh_pattern \.png$ 10080 90% 43200 refresh_pattern \.jpg$ 10080 90% 43200 refresh_pattern \.ico$ 10080 90% 43200 refresh_pattern \.bom\.gov\.au 30 20% 120 refresh_pattern \.html$ 480 50% 22160 refresh_pattern \.htm$ 480 50% 22160 refresh_pattern \.css$ 480 50% 22160 refresh_pattern \.js$ 480 50% 22160 refresh_pattern \.class$ 10080 90% 43200 refresh_pattern \.zip$ 10080 90% 43200 refresh_pattern \.jpeg$ 10080 90% 43200 refresh_pattern \.mid$ 10080 90% 43200 refresh_pattern \.shtml$ 480 50% 22160 refresh_pattern \.exe$ 10080 90% 43200 refresh_pattern \.thm$ 10080 90% 43200 refresh_pattern \.wav$ 10080 90% 43200 refresh_pattern \.mp4$ 10080 90% 43200 refresh_pattern \.txt$ 10080 90% 43200 refresh_pattern \.cab$ 10080 90% 43200 refresh_pattern \.au$ 10080 90% 43200 refresh_pattern \.mov$ 10080 90% 43200 refresh_pattern \.xbm$ 10080 90% 43200 refresh_pattern \.ram$ 10080 90% 43200 refresh_pattern \.iso$ 10080 90% 43200 refresh_pattern \.avi$ 10080 90% 43200 refresh_pattern \.chtml$ 480 50% 22160 refresh_pattern \.thb$ 10080 90% 43200 refresh_pattern \.dcr$ 10080 90% 43200 refresh_pattern \.bmp$ 10080 90% 43200 refresh_pattern \.phtml$ 480 50% 22160 refresh_pattern \.mpg$ 10080 90% 43200 refresh_pattern \.pdf$ 10080 90% 43200 refresh_pattern \.art$ 10080 90% 43200 refresh_pattern \.swf$ 10080 90% 43200 refresh_pattern \.flv$ 10080 90% 43200 refresh_pattern \.x-flv$ 10080 90% 43200 refresh_pattern \.mp3$ 10080 90% 43200 refresh_pattern \.ra$ 10080 90% 43200 refresh_pattern \.spl$ 10080 90% 43200 refresh_pattern \.viv$ 10080 90% 43200 refresh_pattern \.doc$ 10080 90% 43200 refresh_pattern \.gz$ 10080 90% 43200 refresh_pattern \.Z$ 10080 90% 43200 refresh_pattern \.tgz$ 10080 90% 43200 refresh_pattern \.tar$ 10080 90% 43200 refresh_pattern \.vrm$ 10080 90% 43200 refresh_pattern \.vrml$ 10080 90% 43200 refresh_pattern \.aif$ 10080 90% 43200 refresh_pattern \.aifc$ 10080 90% 43200 refresh_pattern \.aiff$ 10080 90% 43200 refresh_pattern \.arj$ 10080 90% 43200 refresh_pattern \.c$ 10080 90% 43200 refresh_pattern \.cpt$ 10080 90% 43200 refresh_pattern \.dir$ 10080 90% 43200 refresh_pattern \.dxr$ 10080 90% 43200 refresh_pattern \.hqx$ 10080 90% 43200 refresh_pattern \.jpe$ 10080 90% 43200 refresh_pattern \.lha$ 10080 90% 43200 refresh_pattern \.lzh$ 10080 90% 43200 refresh_pattern \.midi$ 10080 90% 43200 refresh_pattern \.movie$ 10080 90% 43200 refresh_pattern \.mp2$ 10080 90% 43200 refresh_pattern \.mpe$ 10080 90% 43200 refresh_pattern \.mpeg$ 10080 90% 43200 refresh_pattern \.mpga$ 10080 90% 43200 refresh_pattern \.pl$ 10080 90% 43200 refresh_pattern \.ppt$ 10080 90% 43200 refresh_pattern \.ps$ 10080 90% 43200 refresh_pattern \.qt$ 10080 90% 43200 refresh_pattern \.qtm$ 10080 90% 43200 refresh_pattern \.rar$ 10080 90% 43200 refresh_pattern \.ras$ 10080 90% 43200 refresh_pattern \.sea$ 10080 90% 43200 refresh_pattern \.sit$ 10080 90% 43200 refresh_pattern \.tif$ 10080 90% 43200 refresh_pattern \.tiff$ 10080 90% 43200 refresh_pattern \.snd$ 10080 90% 43200 refresh_pattern \.wrl$ 10080 90% 43200 refresh_pattern ^ftp: 1440 60% 22160 refresh_pattern ^gopher: 1440 20% 1440 refresh_pattern -i (cgi-bin|\?) 0 0% 0 refresh_pattern . 480 50% 22160 # ADMINISTRATIVE PARAMETERS # ----------------------------------------------------------------------------- cache_mgr admin@xxxxxxxxxxx mail_from squid@xxxxxxxxxxx cache_effective_user squid cache_effective_group squid # DELAY POOL PARAMETERS # ----------------------------------------------------------------------------- delay_pools 2 delay_class 1 2 # When big_files are being downloaded, the first 5MB (625000 * 8 bits) are # downloaded at max network speed. Once the file size limit of 5MB is reached, # download speed drops to 438,000 bits or 3,504,000 MB per sec. Current # contracted Internet connection speed w/ TP is at 7MB per sec. delay_parameters 1 750000/750000 438000/625000 acl big_files url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4 .img .flv .wmv .divx .mov .bz2 .deb delay_access 1 allow big_files delay_access 1 deny all delay_class 2 2 # Any files other than big_files are downloaded at wire speed (currently 7MB) # until 6MB file size limit is reached and thereafter at 6MB per sec (750,000 # * 8 bits) delay_parameters 2 750000/750000 750000/750000 delay_access 2 allow localnet delay_access 2 deny big_files delay_access 2 deny all delay_initial_bucket_level 25 # DNS OPTIONS # ----------------------------------------------------------------------------- dns_nameservers 192.168.1.1 append_domain .example.com # MISCELLANEOUS # ----------------------------------------------------------------------------- memory_pools on memory_pools_limit none cachemgr_passwd none all Squid was compiled with the following options: ===> The following configuration options are available for squid-3.4.9: ARP_ACL=on: ARP/MAC/EUI based authentification AUTH_KERB=on: Install Kerberos authentication helpers AUTH_LDAP=on: Install LDAP authentication helpers AUTH_NIS=off: Install NIS/YP authentication helpers AUTH_SASL=off: Install SASL authentication helpers AUTH_SMB=on: Install SMB auth. helpers (req. Samba) AUTH_SQL=off: Install SQL based auth (uses MySQL) CACHE_DIGESTS=on: Use cache digests DEBUG=off: Build with extended debugging support DELAY_POOLS=on: Delay pools (bandwidth limiting) DNS_HELPER=on: Use external dnsserver processes for DNS DOCS=on: Build and/or install documentation ECAP=off: Loadable content adaptation modules ESI=off: ESI support EXAMPLES=on: Build and/or install examples FOLLOW_XFF=on: Support for the X-Following-For header FS_AUFS=on: AUFS (threaded-io) support FS_DISKD=on: DISKD storage engine controlled by separate service FS_ROCK=off: ROCK (unstable) HTCP=on: HTCP support ICAP=off: the ICAP client ICMP=off: ICMP pinging and network measurement IDENT=on: Ident lookups (RFC 931) IPV6=on: IPv6 protocol support KQUEUE=on: Kqueue(2) support LARGEFILE=on: Support large (>2GB) cache and log files LAX_HTTP=off: Do not enforce strict HTTP compliance SNMP=on: SNMP support SSL=on: SSL gatewaying support SSL_CRTD=off: Use ssl_crtd to handle SSL cert requests STACKTRACES=off: Enable automatic backtraces on fatal errors TP_IPF=off: Transparent proxying with IPFilter TP_IPFW=off: Transparent proxying with IPFW TP_PF=off: Transparent proxying with PF VIA_DB=off: Forward/Via database WCCP=on: Web Cache Coordination Protocol WCCPV2=on: Web Cache Coordination Protocol v2 Intially, I set mem_cache=2134MB and after noticing these memory leaks, I dropped it down to 1344MB. Memory leaks are still occurring. Am I using anything that is known to cause memory leaks? If there is additional information that you need, please do not hesitate to ask! Thanks. ~Doug _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
