Landy,
If you are desperate for bandwidth I suggest to block
ads (e.g. a.rad.msn.com) and 'user behaviour analysis'
(e.g. scorecardresearch.com).
Furthermore, you may consider blocking mp3 files.
Depending on what type of users you have, this can save
a lot of bandwidth.
Marcus
Landy Landy wrote:
Hello.
I was looking at a post (how to force windos update to cache all update) from a week ago that was trying to cache all windowsupdates. I was looking into utilizing thundercache that does exactly that, I'm using videocache and can't get both (thundercache and videocache) to work together. After reading the post I decided to use squid to cache windows updates but, don't know if I'm doing it correctly since I haven't copy the refresh_patterns from the post. I actually followed the http://wiki.squid-cache.org/SquidFaq/WindowsUpdate wiki. I would also like to cache antivirus' updates and the content from the most visited sites: hi5, facebook, etc.... I would like to cache everything if possible.
Looking at my access.log file I notice that content I thought it was supposed to be cached is not getting cached for example:
1266160859.409 146 172.16.100.61 TCP_REFRESH_UNMODIFIED/304 382 GET http://col.stb.s-msn.com/i/98/996247A8EF5F5991FCD8AACF6528F.jpg - DIRECT/65.54.81.185 image/jpeg
1266160859.421 148 172.16.100.61 TCP_REFRESH_UNMODIFIED/304 382 GET http://col.stb.s-msn.com/i/FA/E1B3F9B5667878F033D4C68A911AFD.jpg - DIRECT/65.54.81.209 image/jpeg
1266160859.660 168 172.16.100.61 TCP_MISS/200 2273 GET http://a.rad.msn.com/ADSAdClient31.dll? - DIRECT/65.55.197.125 text/html
1266160859.865 100 172.16.100.61 TCP_MISS/200 419 GET http://b.scorecardresearch.com/r? - DIRECT/204.2.241.162 image/gif
1266160861.376 123 172.16.100.61 TCP_REFRESH_UNMODIFIED/304 306 GET http://col.stc.s-msn.com/br/gbl/css/6/decoration/pipe.gif - DIRECT/4.23.59.126 -
1266160861.387 289 172.16.100.45 TCP_MISS/200 415 GET http://0.channel53.facebook.com/p - DIRECT/69.63.178.123 text/plain
1266160862.600 0 172.16.100.45 TCP_MISS/000 0 GET http://0.channel53.facebook.com/p - DIRECT/0.channel53.facebook.com -
1266160865.819 0 172.16.100.18 TCP_MISS/000 0 GET http://sn120w.snt120.mail.live.com/mail/SafeRedirect.aspx? - DIRECT/sn120w.snt120.mail.live.com -
1266160872.391 146 172.16.254.1 TCP_MISS/200 319 GET http://www.kottke.org/frontpage/updates/index.php? - DIRECT/67.18.227.74 text/html
1266160872.647 136 172.16.100.61 TCP_REFRESH_UNMODIFIED/304 264 GET http://col.stb.s-msn.com/i/50/832D93022C9184EBE368DD81A3874.jpg - DIRECT/65.54.81.209 image/jpeg
1266160873.653 346 172.16.100.16 TCP_MISS/200 1119 GET http://www.facebook.com/ajax/presence/reconnect.php? - DIRECT/69.63.189.11 application/x-javascript
1266160874.001 199591 172.16.100.99 TCP_MISS/200 4303656 GET http://streamer.soundclick.com/jarry_lo/14/06/freemp3/mamajuana+ajudemedeus.mp3 - DIRECT/8.14.112.23 audio/x-mpegurl
1266160876.884 270 172.16.100.99 TCP_MISS/200 651 GET http://w88.go.com/b/ss/wdgespcom,wdgespge/1/H.17/s73739908562219? - DIRECT/66.235.138.18 image/gif
1266160877.214 198 172.16.100.110 TCP_MISS/200 5516 GET http://www.myhotcomments.com/graphics/53933 - DIRECT/75.126.132.34 text/html
Here's my squid.conf file. Please correct things that might not be correct or optimized to cache the most content as possible.
# Port Squid listens on
http_port 172.16.0.1:3128 transparent
# Access-lists (ACLs) will permit or deny hosts to access the proxy
acl lan-access src 172.16.0.0/16
acl localhost src 127.0.0.1
acl localnet src 172.16.0.0/16
acl CONNECT method CONNECT
http_access allow localhost
http_access allow lan-access
acl manager proto cache_object
acl webserver src 172.16.0.1
http_access allow manager webserver
http_access deny manager
maximum_object_size 769 MB
cache_mem 1024 MB
access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
cache_dir aufs /var/log/squid/cache 40000 64 255
htcp_port 0
icp_port 0
ignore_unknown_nameservers on
cache_swap_low 90
cache_swap_high 100
max_open_disk_fds 0
memory_replacement_policy heap LRU
###### VideoCache ######
# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/local/videocache/videocache.py
#url_rewrite_program /usr/local/adzap/scripts/zapchain "/usr/bin/python /usr/local/videocache/videocache.py" "/usr/bin/php /usr/local/thundercache/loader.php"
url_rewrite_children 15
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback \.youtube\.com\/videoplay \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.[a-z][a-z]\/videoplayback \.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com .metacafe.com .redtube.com .cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# --END-- videocache config for squid
#---------------------------------
# --- Windows Update
range_offset_limit -1
acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain .go.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com/windowsupdate/v7/default.aspx
acl windowsupdate dstdomain .download.microsoft.com
acl windowsupdate dstdomain activex.microsoft.com
acl windowsupdate dstdomain codecs.microsoft.com
acl windowsupdate dstdomain urs.microsoft.com
#acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow windowsupdate localnet
# --- Windows update ends -----------------------------
store_avg_object_size 48 KB
half_closed_clients off
store_dir_select_algorithm round-robin
quick_abort_min -1
negative_ttl 1 minutes
connect_timeout 90 seconds
dns_nameservers 196.3.81.5 200.88.127.22 196.3.81.132
logfile_rotate 5
offline_mode off
#balance_on_multiple_ip on
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (cgi-bin|\?) 0 0% 0
refresh_pattern . 0 20% 4320
read_ahead_gap 32 KB
visible_hostname Optimum
cache_mgr sdfs@xxxxxxxxxxx
client_persistent_connections off
server_persistent_connections off
persistent_connection_after_error off
detect_broken_pconn off
memory_pools off
#memory_pools_limit 64 MB
refresh_all_ims on
reload_into_ims on
retry_on_error on
coredump_dir none
pipeline_prefetch on
Sorry for the long post but, I'm in desperate need of saving bandwidth since the most I can get in my part of the world is only 5MB and have to handle over 100 users with this connection.
Thanks in advanced for your help and guidance.
