On 28/07/11 19:27, Andrew Rogers wrote:
Hi Amos
Thanks for the detailed reply, I will study your comments accordingly
and will make changes to my squid.conf where necessary thankyou.
On 27 July 2011 07:15, Amos Jeffries<squid3@xxxxxxxxxxxxx> wrote:
This is the sandwich configuration looping on itself. You haev several
choices:
* configure two instances of Squid
* configure client->DG->Squid
* configure client->Squid->DG
* disable "via off" and cross your fingers there is never any actual
infinite loop. You will need a Squid built with HTTP violations enabled to
do that. Loop protection is REQUIRED by the HTTP standards.
I have decided to with a Squid1->DG->Squid2 approach now, and have
already been changing my config files etc accordingly.
This only solves the loop issue though. Cache MISS is separate...
After some initial testing the loop issue has disappeared, and iam
getting quite a few more what looks like cache hits now of
"TCP_REFRESH_UNMODIFIED" not which are showing like:-
1311836509.795 162 localhost TCP_REFRESH_UNMODIFIED/304 553 GET
http://i2.cdnds.net/11/30/P/gaming_sidpirates1.jpg -
DIRECT/213.244.185.38 -
1311836509.795 163 mycomp.tg.local TCP_MISS/304 691 GET
http://i2.cdnds.net/11/30/P/gaming_sidpirates1.jpg me@MY.LOCAL
FIRST_UP_PARENT/127.0.0.1
Which iam assumeing it has had a successfull cache hit from Squid2?
Looks that way. The particular example was a revalidation request
though. If they are both logging to the one file first line is squid2,
second line squid1?
With you saying Cache MISS is seperate, will using the 2 seperate
Squid instances automatically have a better hit rate by the looks
already from here?
It will resolve cases such as ones where a cache HIT prevents the
request going through DG controls of whether the client is allowed that
object.
On the whole, since you have requests that skip DG, and some get cached you
are not in a position to only have caching on the second loop. Things
entering the first loop WILL be found in the cache even if you wanted them
to go through DG first.
Why do some request's skip DG and go direct from the first Squid
instance? I know I have asked whitelistsites to always go direct
which is not a problem, but it is other sites whcih initially go
through Squid->DG but them pages etc which the sites load up seem to
go direct in some circumstances, why is this? and can I configure so
all traffic is forced through DG then to Squid2 instance? If so how
can I achieve this, and what are the knock on effects if any?
Some requests are best served that way rather than going through a
hierarchy. Such as CONNECT requests which are explicit requests to do that.
nonhierarchichal_direct and hierarchy_stoplist control whether these
types of requests are required to go through the peer (DG) or allowed to
go direct.
The effect is a reduction of speed on those requests.
Would something along the line of work:-
http_reply_access allow localhost
http_reply_access deny blockedsites
http_reply_access allow all
always_direct allow localhost
always_direct allow whitelistsites
always_direct deny all
I would want all traffic to go from Squid1->DG and then all traffic
from DG->Squid2 which would then pass all the traffic out direct as
expected?
What can of config along this line would I need in squid2.conf?
http_reply_access is *way* too late to be doing anything like
destination selection. The request has already left squid via some path
and the reply is coming back.
always_direct forces things to go direct. It does not prevent.
never_direct along with the hierarchical controls mentioned above is
what you want to use.
so far this is my changed squid1.conf file:-
####### /etc/squid3/squid.conf Configuration File #######
####### cache manager
cache_mgr squid@xxxxxxxxxxxxxx
####### kerberos authentication
auth_param negotiate program /usr/lib/squid3/squid_kerb_auth -d -s
HTTP/squid.my.local
auth_param negotiate children 10
auth_param negotiate keep_alive on
####### provide access via ldap for clients not authenticated via kerberos
auth_param basic program /usr/lib/squid3/squid_ldap_auth -R \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f sAMAccountName=%s \
-h dc.my.local
auth_param basic children 10
auth_param basic realm Internet Proxy
auth_param basic credentialsttl 1 minute
####### ldap authorizations
# restricted proxy access logged
external_acl_type internet_users %LOGIN /usr/lib/squid3/squid_ldap_group -R -K \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
Users,ou=Internet Groups,dc=my,dc=local))" \
-h dc.my.local
# full proxy access no logging
external_acl_type internet_users_full_nolog %LOGIN
/usr/lib/squid3/squid_ldap_group -R -K \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
Users Full NoLog,ou=Internet Groups,dc=my,dc=local))" \
-h dc.my.local
# full proxy access logged
external_acl_type internet_users_full_log %LOGIN
/usr/lib/squid3/squid_ldap_group -R -K \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
Users Full Log,ou=Internet Groups,dc=my,dc=local))" \
-h dc.my.local
####### acl for proxy auth and ldap authorizations
acl auth proxy_auth REQUIRED
# format "acl, aclname, acltype, acltypename, activedirectorygroup"
acl RestrictedAccessLog external internet_users Internet\ Users
acl FullAccessNoLog external internet_users_full_nolog Internet\
Users\ Full\ NoLog
acl FullAccessLog external internet_users_full_log Internet\ Users\ Full\ Log
acl whitelistsites url_regex -i "/etc/squid3/whitelistsites.txt"
acl blockedsites url_regex -i "/etc/squid3/blockedsites.txt"
##Public DMZ Filtering, do not need to authenticate create acl type
acl DmzPublicWifiSubnet src 10.0.0.0/24
####### squid defaults
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl SSL_ports port 443
acl Safe_ports port 80-81 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
####### enforce auth: order of rules is important for authorization levels
no_cache deny whitelistsites
Remember "no_cache" == "cache"
For no caching you want:
cache deny all
http_access allow whitelistsites
## Allow DMZ Public Wifi Access out without authorisation
http_access allow DmzPublicWifiSubnet
http_access allow FullAccessNoLog auth
http_access allow FullAccessLog auth
http_access deny blockedsites
http_access allow RestrictedAccessLog auth
####### logging
# don't log FullAccessNoLog
access_log /var/log/squid3/access-auth.log squid !FullAccessNoLog
####### squid defaults
http_access deny all
#Log Connecting Client DNS Names instead on IP Names.
log_fqdn on
http_port 8080
##Push Traffic Through DansGuradian for Content Filtering
cache_peer 127.0.0.1 parent 8081 0 no-query proxy-only no-delay
no-netdb-exchange no-digest connect-timeout=15 login=PASS
cache_peer_Access 127.0.0.1 deny localhost
##Push Public DMZ Wifi Subnet through DG
cache_peer_access 127.0.0.1 allow DmzPublicWifiSubnet
cache_peer_access 127.0.0.1 allow RestrictedAccessLog
cache_peer_access 127.0.0.1 deny all
pid_filename /var/run/squid3-auth.pid
visible_hostname mysquid-auth.my.local
hierarchy_stoplist cgi-bin ?
any URL with "?" or "cgi-bin" in it will go DIRECT from this Squid.
Remove "hierarchy_stoplist".
Add these:
nonhierarchical_direct off
to force everything going to DG to ONLY go that way also add these:
never_direct allow DmzPublicWifiSubnet
never_direct allow RestrictedAccessLog
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
squid2.conf:-
####### /etc/squid3/squid.conf Configuration File #######
####### cache manager
cache_mgr squid@xxxxxxxxxxxxx
####### kerberos authentication
auth_param basic program /usr/lib/squid3/squid_ldap_auth -R \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f sAMAccountName=%s \
-h dc.my.local
auth_param basic children 10
auth_param basic realm Internet Proxy
auth_param basic credentialsttl 1 minute
####### ldap authorizations
# restricted proxy access logged
external_acl_type internet_users %LOGIN /usr/lib/squid3/squid_ldap_group -R -K \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
Users,ou=Internet Groups,dc=my,dc=local))" \
-h dc.my.local
# full proxy access no logging
external_acl_type internet_users_full_nolog %LOGIN
/usr/lib/squid3/squid_ldap_group -R -K \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
Users Full NoLog,ou=Internet Groups,dc=my,dc=local))" \
-h dc.my.local
# full proxy access logged
external_acl_type internet_users_full_log %LOGIN
/usr/lib/squid3/squid_ldap_group -R -K \
-b "dc=my,dc=local" \
-D squid@my.local \
-w "password" \
-f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
Users Full Log,ou=Internet Groups,dc=my,dc=local))" \
-h dc.my.local
####### acl for proxy auth and ldap authorizations
acl auth proxy_auth REQUIRED
# format "acl, aclname, acltype, acltypename, activedirectorygroup"
acl RestrictedAccessLog external internet_users Internet\ Users
acl FullAccessNoLog external internet_users_full_nolog Internet\
Users\ Full\ NoLog
acl FullAccessLog external internet_users_full_log Internet\ Users\ Full\ Log
acl whitelistsites url_regex -i "/etc/squid3/whitelistsites.txt"
acl blockedsites url_regex -i "/etc/squid3/blockedsites.txt"
##Public DMZ Filtering, do not need to authenticate create acl type
acl DmzPublicWifiSubnet src 10.0.0.0/24
####### squid defaults
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl SSL_ports port 443
acl Safe_ports port 80-81 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
####### enforce auth: order of rules is important for authorization levels
no_cache deny whitelistsites
http_access allow whitelistsites
## Allow DMZ Public Wifi Access out without authorisation
http_access allow DmzPublicWifiSubnet
http_access allow FullAccessNoLog auth
http_access allow FullAccessLog auth
http_access deny blockedsites
http_access allow RestrictedAccessLog auth
####### logging
# don't log FullAccessNoLog
access_log /var/log/squid3/access-proxy.log squid !FullAccessNoLog
####### squid defaults
http_access deny all
#Log Connecting Client DNS Names instead on IP Names.
log_fqdn on
http_port 127.0.0.1:3128
pid_filename /var/run/squid3-proxy.pid
visible_hostname mysquid-proxy.my.local
hierarchy_stoplist cgi-bin ?
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
End of squid2.conf...
I know I have got duplicate acl's which don't need to be in
squid2.conf, i will remove these shortly, but just need to make sure
iam removing the correct lines which are no longer needed.
How can I make squid1 not cache anything, as I want all the cacheing
to be done at squid2?
see comments above inline with the configs.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.14
Beta testers wanted for 3.2.0.10
