On 17/04/2015 12:51 p.m., iridium191 wrote: > Thanks for your response Amos, it is much appreciated. > The config is below, with comments excluded - we've done tests in the past > to confirm it is not an open proxy and don't believe it is. Any commnts you > may have would also be appreciated. > The past excessive download events correlated with Microsoft patch Tuesdays > or in the most recent case deploying a new Windows server and then manually > updating it, which made us suspect that our refresh rules attempting to > cache Windows updates was the cause of the problem. > > In the config squidguard should be bypassed for Windows updates and > squidclamav uses its own whitelist to bypass Windows update sites. Okay. Noted, and confirmed by the below config. > > Our traffic monitoring so far has been aggregated, so we could see that > 103GB of http traffic was directed to the squid server from the firewall, > and of that 15GB came from Microsoft, 12GB from akamai server 1 etc.. You're > right we didn't consider that something other than squid on the server may > be causing the requests. > Now that you mention clamav ... I had some issues on my own proxies a while back where the freshclam auto-update daemon had partially crashed and on resume was unable to validate the AV updates properly. That pushed it into a loop of re-downloading the entire virus signatures file every few minutes - while the file was only a few dozen KB the constant repeating grew to many GBs over the course of the month before it was caught. > The cache utilization report looks interesting in that we may be able to > script it for more real-time notification of excessive traffic rather than > relying on the morning firewall report. Are there any definitions of the > various counters, eg client_http.kbytes_in, client_http.kbytes_in ? Not that I'm aware of. They should be self-explanatory from the naming though. client_http.kbytes_in -> KB received in to Squid from all clients using HTTP protocol. The section headers explain how long a time period the counters below it cover (5min, 60min, totals since last restart, etc). > > Thanks again, > > acl SSL_ports port 443 > acl Safe_ports port 80 # http > acl Safe_ports port 21 # ftp > acl Safe_ports port 443 # https > acl Safe_ports port 1025-65535 # unregistered ports > acl Safe_ports port 280 # http-mgmt > acl Safe_ports port 488 # gss-http > acl Safe_ports port 591 # filemaker > acl Safe_ports port 777 # multiling http > > acl CONNECT method CONNECT > acl ftp proto FTP > > acl manager url_regex -i ^cache_object:// /squid-internal-mgr/ > acl Purge method PURGE > > acl Local_Networks src 10.250.111.0/24 10.250.112.0/24 > acl BypassCache dst 10.250.111.0/24 10.250.112.0/24 > acl BypassCache dst 146.178.211.0/24 > > acl BypassCacheDomains dstdomain "/etc/squid3/BypassCacheDomains" > acl RestrictedUsers proxy_auth "/etc/squid3/RestrictedUsers" > > # ACLs for Windows Updates & other exceptions > acl WindowsUpdate dstdomain "/etc/squid3/WindowsUpdate" > acl Whitelist_Domains dstdomain "/etc/squid3/Whitelist_Domains" > > # ACL to allow monitoring of entire proxy chain from 10.250.111.124 without > authentication > acl MonitorProxy src 10.250.111.124/32 > > acl Get_Username proxy_auth REQUIRED The above ACL is unused and does nothing. > > # Bypass squidguard for whitelisted domains > redirector_access deny Whitelist_Domains > redirector_access deny WindowsUpdate > # Bypass squidguard for local sites > redirector_access deny BypassCache > redirector_access deny BypassCacheDomains > > # Bypass connections to local network and TLS > always_direct allow BypassCache always_direct does not seem to do what you think it does. All it does is prevent Squid using a cache_peer to service those requests. They are still proxied by *this* Squid. > cache deny BypassCache > always_direct allow BypassCacheDomains > cache deny BypassCacheDomains > > http_access allow manager localhost > http_access allow localhost Purge > http_access deny manager > http_access deny Purge > http_access deny to_localhost > http_access deny !Local_Networks > http_access allow Whitelist_Domains > http_access allow WindowsUpdate > http_access allow MonitorProxy > http_access deny !Safe_ports > http_access deny CONNECT !SSL_ports > # Allow connection to HTTPS sites from the local network > http_access allow CONNECT SSL_ports Local_Networks > http_access allow ftp > http_access allow !RestrictedUsers > > http_access deny all > > http_port 8080 > visible_hostname Squid3 > hierarchy_stoplist cgi-bin ? > > # Log file locations > access_log daemon:/var/log/squid3/access.log squid > cache_store_log none > cache_log /var/log/squid3/cache.log > > # Disk cache directory. > cache_dir aufs /squid_cache/Squid3Cache 25000 16 256 > cache_mem 2000 MB > maximum_object_size_in_memory 1 MB > > # Windows Update > #range_offset_limit 200 MB WindowsUpdate > maximum_object_size 1 GB > #quick_abort_min -1 > > dns_nameservers 127.0.0.1 > > icap_enable on > icap_send_client_ip on > icap_send_client_username on > icap_client_username_encode off > icap_client_username_header X-Authenticated-User > icap_preview_enable on > icap_preview_size 1024 > icap_service service_req reqmod_precache bypass=0 > icap://127.0.0.1:1344/squidclamav > adaptation_access service_req allow all > icap_service service_resp respmod_precache bypass=0 > icap://127.0.0.1:1344/squidclamav > adaptation_access service_resp allow all > > url_rewrite_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf > url_rewrite_children 20 startup=0 idle=1 concurrency=0 > > #Do not show client IP address > via off > forwarded_for off > > #Rules to anonymize http headers > request_header_access Allow allow all > request_header_access Authorization allow all > request_header_access WWW-Authenticate allow all > request_header_access Proxy-Authorization allow all > request_header_access Proxy-Authenticate allow all > request_header_access Content-Encoding allow all > request_header_access Content-Length allow all > request_header_access Content-Type allow all > request_header_access Date allow all > request_header_access Expires allow all > request_header_access Host allow all > request_header_access If-Modified-Since allow all > request_header_access Last-Modified allow all > request_header_access Location allow all > request_header_access Pragma allow all > request_header_access Accept allow all > request_header_access Accept-Charset allow all > request_header_access Accept-Encoding allow all > request_header_access Accept-Language allow all > request_header_access Content-Language allow all > request_header_access Mime-Version allow all > request_header_access Retry-After allow all > request_header_access Title allow all > request_header_access Connection allow all > request_header_access Proxy-Connection allow all > request_header_access Cookie allow all > ###request_header_access All deny all These ones are response-only headers and you can remove from the list: Last-Modified, Location, Retry-After, Date, WWW-Authenticate, Proxy-Authenticate, Expires I recommend adding the Expect, ETag, TE, Transfer-Encoding, If-Match, If-None-Match, If-Unmodified-Since, Range and If-Range headers to the above allow lists. That will allow HTTP/1.1 persistent connections and revalidations to work a lot better. Come to think of it the Range/If-Range and ETag not being allowed is probably related to your problem. You can remove Proxy-Connection. Its an obsolete header Squid does not emit. Mime-Version and Title are also pretty useless unless you have WebDAV clients. Accept-Charset and Accept-Language are not commonly useful and have a large impact on anonymity. Removing them from your allow list could be beneficial. Amos _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
