On 26/08/2014 12:11 p.m., Ragheb Rustom wrote:
> Dear All,
>
> I have lately installed squid 3.3.11 on Centos 6.5 x86_64 system. I have
> configured it as a transparent SSL_BUMP proxy. All is working well I can
> browse all SSL websites successfully after I have imported my generated CA
> file. The problem is that no matter how many times I request the SSL
> websites I always get a TCP_MISS in the squid access log. Among other
> websites I am trying to cache yahoo.com, facebook and youtube but most
> websites are always being served directly from source nothing is being
> served for the squid proxy. Please find below my configuration files. I
> deeply value any help on this matter.
>
For a start configure this and re-check:
strip_query_terms off
That will allow your logs to show the full URL Squid is considering for
cache HIT/MISS. You may find that a few hundred seemingly identical log
entris are in fact highly variable in the query string portion. Such
requests cannot be combined/HIT.
> squid.conf file:
>
> acl snmppublic snmp_community public
> acl bamboe src 10.128.135.0/24
> #uncomment noway url, if necessary.
> #acl noway url_regex -i "/etc/squid/noway"
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 1935 # http acl Safe_ports port 21 #
> ftp acl Safe_ports port 443 # https acl Safe_ports port 70
> # gopher acl Safe_ports port 210 # wais acl Safe_ports port
> 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http acl Safe_ports port 591 #
> filemaker acl Safe_ports port 777 # multiling http
>
>
> acl CONNECT method CONNECT
> #http_access deny noway
> http_access allow manager localhost
> http_access allow bamboe
> http_access deny manager
The above http_access bits...
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
... should be in here.
> http_access allow localhost
> htcp_access deny all
> miss_access allow all
That is the default, you should get faster operation removing
miss_access entirely.
>
> # NETWORK OPTIONS
> http_port 8080
> http_port 8082 intercept
> https_port 8081 intercept ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=8MB cert=/etc/squid/myconfigure.pem
> key=/etc/squid/myconfigure.pem ssl_bump server-first all always_direct allow
> all sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER
>
Avoid DONT_VERIFY_PEER as much as possible. It is "considered harmful"
for security. Also usually unnecessary if the machines trusted CA
certificates are setup properly and up to date.
> sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 8MB
> sslcrtd_children 5 hierarchy_stoplist cgi-bin ? .js .jsp mivo.tv
> 192.168.10.29 192.168.10.30 static.videoku.tv acl QUERY urlpath_regex
> cgi-bin \? .js .jsp 192.168.10.29 192.168.10.30 youtube.com indowebster.com
> static.videoku.tv no_cache deny QUERY
>
Aha!
"no_cache deny QEURY"
The "no_" part is obsolete syntax. What this line actually does is force
all URLs with a query string ('?') to never be cached.
This is the source of your MISS log entries. Remove it to get at least a
chance at some HITs.
Also, hierachy_stoplist is not useful in your configuration. You can
probably remove it entirely. If your squid complains when its missing,
set it to the default:
hierarchy_stoplist /cgi-bin/ \?
> # MEMORY CACHE OPTIONS
> cache_mem 6000 MB
> maximum_object_size_in_memory 16 KB
> memory_replacement_policy heap GDSF
>
> # DISK CACHE OPTIONS
> cache_replacement_policy heap LFUDA
> cache_dir aufs /cache1 300000 64 256
> store_dir_select_algorithm least-load
> minimum_object_size 16 KB
> maximum_object_size 2 GB
Put these global default min/max size limits above the cache_dir lines.
Recent but outdated Squid like yoru 3.3 had a bug where the
maximum_object_size is ignored if configured after cache_dir. Position
for it does not normally matter, so placing it first always works and
avoids needless annoyance.
> cache_swap_low 97
> cache_swap_high 99
>
> #LOGFILE OPTIONS
> access_log stdio:/var/log/squid/access.log cache_log
> /var/log/squid/cache.log cache_store_log none cache_swap_log
> /cache1/swap.state logfile_rotate 5 log_icp_queries off buffered_logs off
>
> #OPTIONS FOR TUNING THE CACHE
Since Squid-3.2 some of the override and ignore options have changed.
* ignore-no-cache is obsolete. Traffic with Cache-Control:no-cache will
be cached properly by default.
- remove this option from your config file.
* combining reload-into-ims and ignore-reload is harmful.
- ignore-reload makes Squid either HIT or MISS, rendering the
revalidate CLIENT_REFRSH performance optimizations enabled by
reload-into-ims useless.
* ignore-private is harmful. Traffic with Cache-Control:private has
mandatory revalidation. What can be cached will be cached properly by
default, this option only causes all private data to be stored - it is
never used from cache.
- remove this option from your config file.
* ignore-auth is harmful. Squid will attempt to cache authenticated
traffic by default. This option *disables* that, and causes the same bad
behaviour as with ignore-private.
(not to mention the legal minefield of privacy, rights and security
issues you are jumping into by trying to force the last two.)
> refresh_pattern -i \.swf$ 20160 80% 20160 override-expire override-lastmod
> reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth
> refresh_pattern -i \.gif$ 20160 80% 20160 override-expire override-lastmod
> reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth
> refresh_pattern -i \.jpg$ 20160 80% 20160 override-expire override-lastmod
> reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth
> refresh_pattern -i \.jpeg$ 20160 80% 20160 override-expire override-lastmod
> reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth
> refresh_pattern -i \.exe$ 20160 80% 20160 override-expire override-lastmod
> reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth
>
> # 1 year = 525600 mins, 1 month = 20160 mins, 1 day = 1440 refresh_pattern
> ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.ad
> brite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xte
> ndmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-a
> dvertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adse
> rving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 20160 20%
> 20160 ignore-no-cache ignore-private override-expire ignore-reload
> ignore-auth refresh_pattern ^.*safebrowsing.*google
> 20160 80% 20160 override-expire ignore-reload ignore-no-cache ignore-private
> ignore-auth refresh_pattern
> ^https://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)
> 20160 80% 20160 override-expire ignore-reload ignore-private refresh_pattern
> ytimg\.com 20160 80%
> 20160 override-expire ignore-reload
> refresh_pattern images\.friendster\.com.*\.(png|gif)
> 20160 80% 20160 override-expire ignore-reload refresh_pattern garena\.com
> 20160 80% 20160 override-expire reload-into-ims refresh_pattern
> photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)
> 20160 80% 20160 override-expire ignore-reload refresh_pattern
> vid\.akm\.dailymotion\.com.*\.on2\?
> 20160 80% 20160 ignore-no-cache override-expire override-lastmod
> refresh_pattern
> mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 20160 80%
> 20160 reload-into-ims override-expire ignore-private refresh_pattern
> ^http:\/\/images|pics|thumbs[0-9]\.
> 20160 80% 20160 reload-into-ims ignore-no-cache ignore-reload
> override-expire refresh_pattern ^http:\/\/www.onemanga.com.*\/
> 20160 80% 20160 reload-into-ims ignore-no-cache ignore-reload
> override-expire refresh_pattern
> ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9])
> 20160 80% 20160 override-expire ignore-reload ignore-no-cache ignore-private
> ignore-auth override-lastmod #images facebook refresh_pattern -i
> \.facebook.com 20160 80% 20160 ignore-reload
> override-expire ignore-no-cache
>
> # Facebook
> refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif|css)
> 20160 80% 20160 ignore-reload override-expire ignore-no-cache
> refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)
> 20160 80% 20160 ignore-reload override-expire ignore-no-cache
Note how the regext pattern above matches the same URLs as all the below
patterns. This means the below ones will *never be used*.
> refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png)
> 20160 80% 20160 ignore-reload override-expire ignore-no-cache
> refresh_pattern ^https:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)
> 20160 80% 20160 ignore-reload override-expire ignore-no-cache
> refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)
> 20160 80% 20160 ignore-reload override-expire ignore-no-cache
> refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png)
> 20160 80% 20160 ignore-reload override-expire ignore-no-cache
> refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)
> 20160 80% 20160 ignore-reload override-expire ignore-no-cache
>
Also, FB is actually one of the cache friendly providers nowdays. You
may want to re-assess whether overriding their headers is useful or harmful.
> #All File
> refresh_pattern -i
> \.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt) 20160
> 80% 20160 ignore-no-cache override-expire override-lastmod reload-into-ims
> refresh_pattern -i
> \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar) 20160
> 80% 20160 ignore-no-cache override-expire override-lastmod reload-into-ims
> refresh_pattern -i
> \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll) 20160
> 80% 20160 ignore-no-cache override-expire override-lastmod reload-into-ims
> refresh_pattern -i
> \.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|sn
> d|vob) 20160 80% 20160 ignore-no-cache override-expire
> d|override-lastmod
> reload-into-ims
> refresh_pattern -i
> \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|
> x-flv) 20160 80% 20160 ignore-no-cache override-expire override-lastmod
> reload-into-ims refresh_pattern ^ftp: 1440 90% 201600
> override-lastmod reload-into-ims refresh_pattern ^gopher: 1440
> 0% 1440 override-lastmod reload-into-ims refresh_pattern
> (cgi-bin|\?) 0 0% 0 refresh_pattern . 1440
> 80% 201600 override-lastmod reload-into-ims quick_abort_min 64 KB
> quick_abort_max 64 KB quick_abort_pct 95
>
> shutdown_lifetime 10 seconds
> half_closed_clients off
> cache_effective_user squid
> cache_effective_group squid
>
> dns_nameservers 127.0.0.1 46.20.98.62 8.8.8.8 8.8.4.4 ipcache_size 2048
> ipcache_low 90 ipcache_high 95
>
> #another optimizing
> memory_pools off
> client_db on
> coredump_dir /cache1
> reload_into_ims on
> balance_on_multiple_ip on
balance_on_multiple_ip is kind of harmful for user experience in the
modern Internet. There are far too many web applications and network
load balancing systems out there which rely on the browser behaviour of
sticking to one server IP for all traffic in a browsing session.
> vary_ignore_expire on
> pipeline_prefetch on
> max_filedescriptors 65535
>
> #MARKING ZPH for squid 3.1
> qos_flows local-hit=0x30
>
Please do not forget to run "squid -k check" after editing your config
file. Many of these issues would have been notified to you by Squid itself.
Amos
