On 22/07/11 16:00, Wilson Hernandez wrote:
On 7/20/2011 9:12 PM, Amos Jeffries wrote:
On Wed, 20 Jul 2011 11:12:04 -0400, Wilson Hernandez wrote:
Hello.
I am puzzled to see how my bandwidth is used when running squid. I have
a total of 25M/3M of bandwidth, lately I've noticed with iptraf that my
external interface traffic/bandwidth is almost maxed out at 24.8M and my
internal interface (squid) is only at 2.9M as a result most clients have
been calling saying "their internet is slow".
I'm wondering why that big of a difference on the interfaces' traffic.
This is what cachemgr shows:
Squid Object Cache: Version 3.1.14
Start Time: Fri, 15 Jul 2011 08:01:48 GMT
Current Time: Wed, 20 Jul 2011 14:39:02 GMT
Connection information for squid:
Number of clients accessing cache: 113
Number of HTTP requests received: 5198204
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 684.2
Average ICP messages per minute since start: 0.0
Select loop called: 479758718 times, 0.950 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 23.2%, 60min: 19.4%
Hits as % of bytes sent: 5min: -219.3%, 60min: -314.7%
Memory hits as % of hit requests: 5min: 13.2%, 60min: 9.5%
Disk hits as % of hit requests: 5min: 64.6%, 60min: 62.5%
Storage Swap size: 66028580 KB
Storage Swap capacity: 64.5% used, 35.5% free
Storage Mem size: 1042556 KB
Storage Mem capacity: 100.0% used, 0.0% free
Mean Object Size: 23.52 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.12106 0.02069
Cache Misses: 0.24524 0.30459
Cache Hits: 0.05046 0.02899
Near Hits: 0.17711 0.22004
Not-Modified Replies: 0.00307 0.00091
DNS Lookups: 0.31806 0.17048
DNS is very slow as well. Probably due to remote queries over this
full link.
Please help me understand why this is happening and if there is a
solution to make squid perform better.
Squid "optimizes web delivery" as the slogan goes. So when the server
side is acting very inefficiently it can consume more than the client
side. Could be any of these or a few other things I'm not aware of:
1) client requests an object. Squid has it cached, but server is
requiring 'must-revalidate'. While revalidating the server forces an
entire new object back at squid, along with a timestamp stating it has
not changed. Squid only sends the small no-change reply to the client.
2a) client requests a small range of an object. Squid passes this on.
Server replies with again, forcing an entire new object back at squid.
Squid only sends the small range asked for to the client.
2b) client requests a small range of an object. Squid passes this on
but requests the full object (refresh_offset_limit). Server replies
with the whole object. Squid stores it and only sends the small range
asked for to the client.
Sorry, mistake. 'refresh_offset_limit' should have been 'range_offset_limit.
3) client requests info about an object (HEAD). Squid relays this
request on. Server replies, forcing an entire new object back at
squid. Squid only sends the small header asked for to the client.
4) client requests an object, then abandons it before receiving the
reply. Squid continues to wait and receive it, in hopes that it can be
stored. If not storable it may be discarded and the cycle repeat. Or
it could be stored but never again requested. This behaviour is
modified by the quick_abort_* directives.
Or it could be you configured an open proxy. Configuration problems
can allow external access to external sites. When discovered attackers
can use this and consume all your external bandwidth. Usually its
caused by mistakenly removing or bypassing the controls on CONNECT
tunnels. Though it can also happen on other requests.
Amos
Amos.
Thanks for replying. Now, you have left me confused and in doubt. I
don't know if my actual configuration is ok or not. I will post it here
so, you can take a look and let me know where or what I'm doing wrong.
Thanks again.
Sorry. "slow" is a vague problem :(
squid.conf:
# Port Squid listens on
http_port 172.16.0.1:3128 intercept disable-pmtu-discovery=off
Port 3128 is very dangerous for interception. It is best to pick a
completely random one.
Then to firewall any access to it. These ports are _only_ used between
squid and its box NAT system. Some hints on locking it down (iptabes
mangle table) are on the wiki intercept pages
http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxDnat
error_default_language es-do
# Access-lists (ACLs) will permit or deny hosts to access the proxy
acl lan-access src 172.16.0.0/16
acl localhost src 127.0.0.1
acl localnet src 172.16.0.0/16
lan-access and localnet look like the same thing. You can simplify be
removing one.
acl proxy src 172.16.0.1
acl clientes_registrados src "/etc/msd/ipAllowed"
acl adstoblock dstdomain "/etc/squid/blockAds"
acl CONNECT method CONNECT
<snip comments>
#-----------------------
http_access allow proxy
http_access allow localhost
What is "172.16.0.1" and localhost doing having completely unrestricted
access?
#---- Block some sites
acl blockanalysis01 dstdomain .scorecardresearch.com clkads.com
acl blockads01 dstdomain .rad.msn.com ads1.msn.com ads2.msn.com
ads3.msn.com ads4.msn.com
acl blockads02 dstdomain .adserver.yahoo.com
pagead2.googlesyndication.com ad.yieldmanager.com
acl blockads03 dstdomain .doubleclick.net .fastclick.net
.googleadservices.com
acl blockads04 dstdomain .ero-advertising.com .adsomega.com
acl blockads05 dstdomain .adyieldmanager.com .yieldmanager.com
.adyieldmanager.net .yieldmanager.net
acl blockads06 dstdomain .e-planning.net .super-publicidad.com
.super-publicidad.net
acl blockads07 dstdomain .adbrite.com .contextweb.com .adbasket.net
.clicktale.net
acl blockads08 dstdomain .adserver.com .adv-adserver.com
.zerobypass.info .zerobypass.com
acl blockads09 dstdomain ads.ak.facebook.com .pubmatic.com .baynote.net
.publicbt.com
http_access deny blockanalysis01
http_access deny blockads01
http_access deny blockads02
http_access deny blockads03
http_access deny blockads04
http_access deny blockads05
http_access deny blockads06
http_access deny blockads07
http_access deny blockads08
http_access deny blockads09
http_access deny adstoblock
All of these block lists can be combined into one ACL name. ie
"blockads". Which will be much faster to check and block with.
acl bank dstdomain .popularenlinea.com .bpd.com.do .google.com
.google.com.do
google.com is your _bank_? wow.
acl ourPublicServer dst 190.80.159.7
http_access allow bank
http_access allow ourPublicServer
Attempt at a reverse-proxy using squid-2.5 method? you may want to take
a look at how this is done with modern Squid.
http://wiki.squid-cache.org/ConfigExamples/Reverse/BasicAccelerator
http://wiki.squid-cache.org/ConfigExamples/Reverse/VirtualHosting
This is one possible cause of "slow". Almost every single request has to
do DNS lookups to figure out if its going to 190.80.159.7 or not.
Surely you have one/some domain names used to access that server?
Adding a dstdomain list of those will speed things up quite a bit.
Moving to the reverse-proxy config for that same list of domains will
speed these particular requests even more.
<snip comments>
acl manager proto cache_object
# replace 10.0.0.1 with your webserver IP
acl webserver src 172.16.0.1
http_access allow manager webserver
http_access deny manager
You called 172.16.0.1 "proxy" earlier and gave it unlimited access.
This management access security is one of the things which should be at
the top of your http_access sequence.
http://wiki.squid-cache.org/SquidFaq/SecurityPitfalls#The_manager_ACLs
<snip comments>
#access_log /var2/squid/access.log
access_log none
It would be a good idea to enable logging for a while and see what type
of requests are going through the proxy. They are likely to be a big
clue about the imbalance.
<snip comments>
# Also videos are LARGE; make sure you aren't killing them as 'too big
to save'
# - squid defaults to 4MB, which is too small for videos and even some
sound files
maximum_object_size 32500 KB
>
> maximum_object_size_in_memory 15625 KB
Perhapse it would be easier to write this:
maximum_object_size 32 MB
maximum_object_size_in_memory 16 MB
<snip>
acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain .go.microsoft.com
acl windowsupdate dstdomain
.update.microsoft.com/windowsupdate/v7/default.aspx
acl windowsupdate dstdomain .download.microsoft.com
acl windowsupdate dstdomain activex.microsoft.com
acl windowsupdate dstdomain codecs.microsoft.com
acl windowsupdate dstdomain urs.microsoft.com
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow windowsupdate localnet
# --- Windows update ends -----------------------------
# ------ Test AntiVirus Caching --------------
acl avast_allow_url url_regex -i \.vpu
acl avast_allow_url url_regex -i \.vpx
url_rewrite_access allow avast_allow_url
acl avast dstdomain avast.com
http_access allow CONNECT localnet
I notice this is the last mention of CONNECT. Nowhere to be seen is the
safety control preventing internal viral infections from using the proxy.
Please read:
http://wiki.squid-cache.org/SquidFaq/SecurityPitfalls#The_Safe_Ports_and_SSL_Ports_ACL
http_access allow avast localnet
<snip comments and default options>
range_offset_limit 0 KB
quick_abort_min 0 KB
Okay. That prevents scenario (4) from happening. The side effect is
larer total bandwidth consued if clients about, but not an imbalance
like you are seeing.
#quick_abort_pct 95
range_offset_limit -1
This line replaces the earlier range_offset_limit of 0.
This one may cause scenario (2b) I mentioned above.
<snip>
balance_on_multiple_ip on
This is not such a good idea. It breaks persistent connections,
IPv4/Ipv6 gateway and any websites which assume the client session will
only go to one of their servers.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
The "." pattern matches absolutely every URL that can possible exist.
Squid will never even attempt to evaluate the patterns which follow that
rule.
If you find that things have been working okay, then you can erase every
refresh_pattern below this line.
#Suggested default:
refresh_pattern -i \.jpg$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.gif$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.png$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.jpeg$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.bmp$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.tif$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.tiff$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.swf$ 0 50% 21600 reload-into-ims
refresh_pattern -i \.html$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.htm$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.shtml$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.shtm$ 0 20% 1440 reload-into-ims
refresh_pattern -i \.nub$ 2880 80% 21600 reload-into-ims
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 8640
refresh_pattern -i exe$ 0 50% 525600
refresh_pattern -i zip$ 0 50% 525600
refresh_pattern -i \.flv$ 10080 90% 525600 ignore-no-cache
override-expire ignore-private
refresh_pattern -i \.swf$ 10080 90% 525600 ignore-no-cache
override-expire ignore-private
read_ahead_gap 32 KB
visible_hostname Optimum-Wireless-Services
Funny looking domain name.
http://Optimum-Wireless-Services/ doesn't go anywhere useful for me.
<snip comments>
#Default:
client_persistent_connections on
server_persistent_connections on
But, but, you also have "balance_on_mutiple_ip on" Which means
round-robin the destination IPs. This will raise the number of
connections you have, eroding benefits from the persistence.
<snip>
http_access allow clientes_registradosr
shutdown_lifetime 45 seconds
http_access deny all
Amos
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.14
Beta testers wanted for 3.2.0.9
