On 7/20/2011 9:12 PM, Amos Jeffries wrote:
On Wed, 20 Jul 2011 11:12:04 -0400, Wilson Hernandez wrote:
Hello.
I am puzzled to see how my bandwidth is used when running squid. I have
a total of 25M/3M of bandwidth, lately I've noticed with iptraf that my
external interface traffic/bandwidth is almost maxed out at 24.8M and my
internal interface (squid) is only at 2.9M as a result most clients have
been calling saying "their internet is slow".
I'm wondering why that big of a difference on the interfaces' traffic.
This is what cachemgr shows:
Squid Object Cache: Version 3.1.14
Start Time: Fri, 15 Jul 2011 08:01:48 GMT
Current Time: Wed, 20 Jul 2011 14:39:02 GMT
Connection information for squid:
Number of clients accessing cache: 113
Number of HTTP requests received: 5198204
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 684.2
Average ICP messages per minute since start: 0.0
Select loop called: 479758718 times, 0.950 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 23.2%, 60min: 19.4%
Hits as % of bytes sent: 5min: -219.3%, 60min: -314.7%
Memory hits as % of hit requests: 5min: 13.2%, 60min: 9.5%
Disk hits as % of hit requests: 5min: 64.6%, 60min: 62.5%
Storage Swap size: 66028580 KB
Storage Swap capacity: 64.5% used, 35.5% free
Storage Mem size: 1042556 KB
Storage Mem capacity: 100.0% used, 0.0% free
Mean Object Size: 23.52 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.12106 0.02069
Cache Misses: 0.24524 0.30459
Cache Hits: 0.05046 0.02899
Near Hits: 0.17711 0.22004
Not-Modified Replies: 0.00307 0.00091
DNS Lookups: 0.31806 0.17048
DNS is very slow as well. Probably due to remote queries over this
full link.
Please help me understand why this is happening and if there is a
solution to make squid perform better.
Squid "optimizes web delivery" as the slogan goes. So when the server
side is acting very inefficiently it can consume more than the client
side. Could be any of these or a few other things I'm not aware of:
1) client requests an object. Squid has it cached, but server is
requiring 'must-revalidate'. While revalidating the server forces an
entire new object back at squid, along with a timestamp stating it has
not changed. Squid only sends the small no-change reply to the client.
2a) client requests a small range of an object. Squid passes this on.
Server replies with again, forcing an entire new object back at squid.
Squid only sends the small range asked for to the client.
2b) client requests a small range of an object. Squid passes this on
but requests the full object (refresh_offset_limit). Server replies
with the whole object. Squid stores it and only sends the small range
asked for to the client.
3) client requests info about an object (HEAD). Squid relays this
request on. Server replies, forcing an entire new object back at
squid. Squid only sends the small header asked for to the client.
4) client requests an object, then abandons it before receiving the
reply. Squid continues to wait and receive it, in hopes that it can be
stored. If not storable it may be discarded and the cycle repeat. Or
it could be stored but never again requested. This behaviour is
modified by the quick_abort_* directives.
Or it could be you configured an open proxy. Configuration problems
can allow external access to external sites. When discovered attackers
can use this and consume all your external bandwidth. Usually its
caused by mistakenly removing or bypassing the controls on CONNECT
tunnels. Though it can also happen on other requests.
Amos
Amos.
Thanks for replying. Now, you have left me confused and in doubt. I
don't know if my actual configuration is ok or not. I will post it here
so, you can take a look and let me know where or what I'm doing wrong.
Thanks again.
squid.conf:
# Port Squid listens on
http_port 172.16.0.1:3128 intercept disable-pmtu-discovery=off
error_default_language es-do
# Access-lists (ACLs) will permit or deny hosts to access the proxy
acl lan-access src 172.16.0.0/16
acl localhost src 127.0.0.1
acl localnet src 172.16.0.0/16
acl proxy src 172.16.0.1
acl clientes_registrados src "/etc/msd/ipAllowed"
acl adstoblock dstdomain "/etc/squid/blockAds"
acl CONNECT method CONNECT
#---- Do not cache these
#acl special_domains dstdomain .facebook.com .fbcdn.net .verisign.com
.mail.yahoo.com
#cache deny special_domains
#-----------------------
http_access allow proxy
http_access allow localhost
#---- Block some sites
acl blockanalysis01 dstdomain .scorecardresearch.com clkads.com
acl blockads01 dstdomain .rad.msn.com ads1.msn.com ads2.msn.com
ads3.msn.com ads4.msn.com
acl blockads02 dstdomain .adserver.yahoo.com
pagead2.googlesyndication.com ad.yieldmanager.com
acl blockads03 dstdomain .doubleclick.net .fastclick.net
.googleadservices.com
acl blockads04 dstdomain .ero-advertising.com .adsomega.com
acl blockads05 dstdomain .adyieldmanager.com .yieldmanager.com
.adyieldmanager.net .yieldmanager.net
acl blockads06 dstdomain .e-planning.net .super-publicidad.com
.super-publicidad.net
acl blockads07 dstdomain .adbrite.com .contextweb.com
.adbasket.net .clicktale.net
acl blockads08 dstdomain .adserver.com .adv-adserver.com
.zerobypass.info .zerobypass.com
acl blockads09 dstdomain ads.ak.facebook.com .pubmatic.com
.baynote.net .publicbt.com
http_access deny blockanalysis01
http_access deny blockads01
http_access deny blockads02
http_access deny blockads03
http_access deny blockads04
http_access deny blockads05
http_access deny blockads06
http_access deny blockads07
http_access deny blockads08
http_access deny blockads09
http_access deny adstoblock
acl bank dstdomain .popularenlinea.com .bpd.com.do .google.com
.google.com.do
acl ourPublicServer dst 190.80.159.7
http_access allow bank
http_access allow ourPublicServer
#acl block url_regex "/etc/squid/sitesblocked"
#http_access deny block
#---- End block sites
# Access rule
#http_access allow clientes_registrados
acl manager proto cache_object
# replace 10.0.0.1 with your webserver IP
acl webserver src 172.16.0.1
http_access allow manager webserver
http_access deny manager
########################## Delay pools ####################
# Replace the network below with your own
# Define 1 delay pool, class 2
#delay_pools 1
#delay_class 1 3
# Manage traffic from our network with the delay pool
#delay_access 1 allow clientes_registrados
#delay_access 1 deny all
# Values are in bytes, to get kbps multiply by 8
#delay_parameters 1 1250000/1250000 1000000/1000000 64000/64000
############################################################
# Block Malware Section
# File which contains the list
# acl malware_block_list url_regex -i "/etc/squid/malware_block_list.txt"
# Access Denied
# http_access deny malware_block_list
# Redirect message - (You can make your own)
# deny_info http://malware.hiperlinks.com.br/denied.shtml malware_block_list
# Also videos are LARGE; make sure you aren't killing them as 'too big
to save'
# - squid defaults to 4MB, which is too small for videos and even some
sound files
maximum_object_size 32500 KB
maximum_object_size_in_memory 15625 KB
minimum_object_size 0 KB
cache_mem 1024 MB
#access_log /var2/squid/access.log
access_log none
cache_log /var2/squid/cache.log
cache_store_log none
#cache_dir ufs /var/log/squid/cache 5000 16 256
#cache_dir ufs /var/log/squid/cache 40000 255 255
#cache_dir aufs /var/log/squid/cache 140000 64 255
cache_dir aufs /var2/squid/cache 100000 64 255
#cache_dir aufs /squidtest 5000 16 16
htcp_port 0
icp_port 0
ignore_unknown_nameservers on
cache_swap_low 97
cache_swap_high 99
max_open_disk_fds 0
# httpd_suppress_version_string
#hierarchy_stoplist cgi-bin ?
#acl QUERY urlpath_regex cgi-bin \?
#cache deny QUERY
#acl GOOGLEVIDEO urlpath_regex /videoplayback\?id= /get_video\?origin=
#cache deny GOOGLEVIDEO
#acl YOUTUBE urlpath_regex /get_video\?video_id=
#cache deny YOUTUBE
#acl CACHE_HOST_IP dst 172.16.0.1
#cache deny CACHE_HOST_IP
memory_replacement_policy heap LRU
###### ecap gzip section ######
#ecap_enable on
#ecap_service gzip_service respmod_precache 0 ecap://www.vigos.com/ecap_gzip
#loadable_modules /usr/local/lib/ecap_adapter_gzip.so
#acl GZIP_HTTP_STATUS http_status 200
#adaptation_access gzip_service allow GZIP_HTTP_STATUS
################################
###### Newer VideoCache ######
# --BEGIN-- videocache config for squid
#url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
#url_rewrite_program /usr/bin/python2.5 /usr/local/videocache/videocache.py
#url_rewrite_program /usr/local/adzap/scripts/zapchain "/usr/bin/python
/usr/share/videocache/videocache.py" "/usr/bin/php
/usr/local/thundercache/loader.php"
url_rewrite_program /usr/local/bin/zapchain "/usr/bin/python
/usr/share/videocache/videocache.py" "/usr/local/bin/squidGuard -c
/usr/local/squidGuard/squidGuard.conf"
#url_rewrite_program /usr/local/bin/wrapzap
url_rewrite_children 10
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback
\.youtube\.com\/videoplay \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i
\.youtube\.[a-z][a-z]\/videoplayback \.youtube\.[a-z][a-z]\/videoplay
\.youtube\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback
\.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback
\.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i
\.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay
\.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i
proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i
[a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i
va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i
media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i
\.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com .metacafe.com
.redtube.com .cdn.dailymotion.com
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
url_rewrite_access allow all
redirector_bypass on
# --END-- videocache config for squid
#---------------------------------
# --- Windows Update
acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain .go.microsoft.com
acl windowsupdate dstdomain
.update.microsoft.com/windowsupdate/v7/default.aspx
acl windowsupdate dstdomain .download.microsoft.com
acl windowsupdate dstdomain activex.microsoft.com
acl windowsupdate dstdomain codecs.microsoft.com
acl windowsupdate dstdomain urs.microsoft.com
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow windowsupdate localnet
# --- Windows update ends -----------------------------
# ------ Test AntiVirus Caching --------------
acl avast_allow_url url_regex -i \.vpu
acl avast_allow_url url_regex -i \.vpx
url_rewrite_access allow avast_allow_url
acl avast dstdomain avast.com
http_access allow CONNECT localnet
http_access allow avast localnet
#---------------------------------
#url_rewrite_children 10
#acl store_rewrite_list url_regex -i "/etc/thundercache/thunder.lst"
#url_rewrite_access allow store_rewrite_list
#url_rewrite_access deny all
#url_rewrite_program /usr/local/thundercache/loader.php
#---------------------------------
#===================================================================#
#Redirecionamento Thunder 3.x - REGEx
#===================================================================#
#acl thunder_lst url_regex -i "/etc/thundercache/thunder.lst"
#cache deny thunder_lst
#cache_peer 172.16.0.1 parent 8000 0 proxy-only no-digest
#dead_peer_timeout 2 seconds
#cache_peer_access 172.16.0.1 allow thunder_lst
#cache_peer_access 172.16.0.1 deny all
#==================================================================#
# TAG: store_avg_object_size (kbytes)
# Average object size, used to estimate number of objects your
# cache can hold. The default is 13 KB.
#
#Default:
store_avg_object_size 64 KB
# TAG: half_closed_clients
# Some clients may shutdown the sending side of their TCP
# connections, while leaving their receiving sides open. Sometimes,
# Squid can not tell the difference between a half-closed and a
# fully-closed TCP connection. By default, half-closed client
# connections are kept open until a read(2) or write(2) on the
# socket returns an error. Change this option to 'off' and Squid
# will immediately close client connections when read(2) returns
# "no more data to read."
#
#Default:
half_closed_clients off
#Default:
store_dir_select_algorithm least-load
#extension_methods SEARCH NICK
range_offset_limit 0 KB
quick_abort_min 0 KB
#quick_abort_pct 95
range_offset_limit -1
#negative_ttl 1 minutes
connect_timeout 60 seconds
dns_nameservers 172.16.0.2 172.16.0.1
logfile_rotate 5
offline_mode off
balance_on_multiple_ip on
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
#Suggested default:
refresh_pattern -i \.jpg$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.gif$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.png$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.jpeg$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.bmp$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.tif$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.tiff$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.swf$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.html$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.htm$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.shtml$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.shtm$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.nub$ 2880 80% 21600 reload-into-ims
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 8640
refresh_pattern -i exe$ 0 50% 525600
refresh_pattern -i zip$ 0 50% 525600
refresh_pattern -i \.flv$ 10080 90% 525600 ignore-no-cache
override-expire ignore-private
refresh_pattern -i \.swf$ 10080 90% 525600 ignore-no-cache
override-expire ignore-private
read_ahead_gap 32 KB
visible_hostname Optimum-Wireless-Services
cache_mgr optimumwireless@xxxxxxxxxxx
# TAG: store_dir_select_algorithm
# Set this to 'round-robin' as an alternative.
#
#Default:
# store_dir_select_algorithm least-load
store_dir_select_algorithm round-robin
# PERSISTENT CONNECTION HANDLING
#
-----------------------------------------------------------------------------
#
# Also see "pconn_timeout" in the TIMEOUTS section
# TAG: client_persistent_connections
# TAG: server_persistent_connections
# Persistent connection support for clients and servers. By
# default, Squid uses persistent connections (when allowed)
# with its clients and servers. You can use these options to
# disable persistent connections with clients and/or servers.
#
#Default:
client_persistent_connections on
server_persistent_connections on
# TAG: persistent_connection_after_error
# With this directive the use of persistent connections after
# HTTP errors can be disabled. Useful if you have clients
# who fail to handle errors on persistent connections proper.
#
#Default:
persistent_connection_after_error off
# TAG: detect_broken_pconn
# Some servers have been found to incorrectly signal the use
# of HTTP/1.0 persistent connections even on replies not
# compatible, causing significant delays. This server problem
# has mostly been seen on redirects.
#
# By enabling this directive Squid attempts to detect such
# broken replies and automatically assume the reply is finished
# after 10 seconds timeout.
#
#Default:
detect_broken_pconn off
# TAG: memory_pools on|off
# If set, Squid will keep pools of allocated (but unused) memory
# available for future use. If memory is a premium on your
# system and you believe your malloc library outperforms Squid
# routines, disable this.
#
#Default:
memory_pools on
# TAG: memory_pools_limit (bytes)
# Used only with memory_pools on:
# memory_pools_limit 50 MB
#
# If set to a non-zero value, Squid will keep at most the specified
# limit of allocated (but unused) memory in memory pools. All free()
# requests that exceed this limit will be handled by your malloc
# library. Squid does not pre-allocate any memory, just safe-keeps
# objects that otherwise would be free()d. Thus, it is safe to set
# memory_pools_limit to a reasonably high value even if your
# configuration will use less memory.
#
# If set to zero, Squid will keep all memory it can. That is, there
# will be no limit on the total amount of memory used for safe-keeping.
#
# To disable memory allocation optimization, do not set
# memory_pools_limit to 0. Set memory_pools to "off" instead.
#
# An overhead for maintaining memory pools is not taken into account
# when the limit is checked. This overhead is close to four bytes per
# object kept. However, pools may actually _save_ memory because of
# reduced memory thrashing in your malloc library.
#
#Default:
memory_pools_limit 64 MB
# TAG: refresh_all_ims on|off
# When you enable this option, squid will always check
# the origin server for an update when a client sends an
# If-Modified-Since request. Many browsers use IMS
# requests when the user requests a reload, and this
# ensures those clients receive the latest version.
#
# By default (off), squid may return a Not Modified response
# based on the age of the cached version.
#
#Default:
refresh_all_ims off
# TAG: reload_into_ims on|off
# When you enable this option, client no-cache or ``reload''
# requests will be changed to If-Modified-Since requests.
# Doing this VIOLATES the HTTP standard. Enabling this
# feature could make you liable for problems which it
# causes.
#
# see also refresh_pattern for a more selective approach.
#
#Default:
reload_into_ims off
# TAG: retry_on_error
# If set to on Squid will automatically retry requests when
# receiving an error response. This is mainly useful if you
# are in a complex cache hierarchy to work around access
# control errors.
#
#Default:
retry_on_error on
# TAG: coredump_dir
# By default Squid leaves core files in the directory from where
# it was started. If you set 'coredump_dir' to a directory
# that exists, Squid will chdir() to that directory at startup
# and coredump files will be left there.
#
#Default:
# coredump_dir none
#
# Leave coredumps in the first cache dir
coredump_dir none
# TAG: pipeline_prefetch
# To boost the performance of pipelined requests to closer
# match that of a non-proxied environment Squid can try to fetch
# up to two requests in parallel from a pipeline.
#
# Defaults to off for bandwidth management and access logging
# reasons.
#
#Default:
pipeline_prefetch on
http_access allow clientes_registrados
shutdown_lifetime 45 seconds
http_access deny all
