On 1/18/2010 8:55 PM, Amos Jeffries wrote:
On Mon, 18 Jan 2010 13:18:20 -0500, Jason Spegal<jspegal@xxxxxxxxxxx>
wrote:
Alrighty. Did some more research and found a solution to my problem
which leads to another issue.
My problem: I was trying to serve a proxy auto configuration file
(wpad.dat) from an internal webserver (http://wpad/). When the client
down the pipe after squid picked it up the file was served with the mime
type chemical/x-mopac-input. When I went direct to the webserver it
served the correct mime type (which I had forced it to).
Solution: On Gentoo squid is using the /etc/mime.types file to guess the
mime type instead of what the remote webserver is saying the file is. I
Point 1: Squid does not do that. Does not use mime.types at all.
Content-Type headers are passed through unchanged from what is received
unless administratively changed by header_replace.
Taken from access.log
Before changing mime.types
1263657638.249 0 10.10.122.248 TCP_MEM_HIT/200 670 GET
http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
1263661679.834 0 10.10.122.239 TCP_MEM_HIT/200 670 GET
http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
1263662648.054 9 10.10.122.248 TCP_CLIENT_REFRESH_MISS/200 654 GET
http://wpad/wpad.dat - DIRECT/10.10.122.250 chemical/x-mopac-input
1263662742.482 4 10.10.122.248 TCP_CLIENT_REFRESH_MISS/200 654 GET
http://wpad/wpad.dat - DIRECT/10.10.122.250 chemical/x-mopac-input
1263662752.973 0 10.10.122.248 TCP_IMS_HIT/304 264 GET
http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
1263664740.203 0 10.10.122.248 TCP_MEM_HIT/200 669 GET
http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
After changing mime.types
1263834369.649 1 10.10.122.241 TCP_REFRESH_UNMODIFIED/200 647 GET
http://wpad/wpad.dat - DIRECT/10.10.122.250
application/x-ns-proxy-autoconfig
1263834539.719 0 10.10.122.241 TCP_MEM_HIT/200 657 GET
http://wpad/wpad.dat - NONE/- application/x-ns-proxy-autoconfig
1263834791.576 0 10.10.122.241 TCP_MEM_HIT/200 657 GET
http://wpad/wpad.dat - NONE/- application/x-ns-proxy-autoconfig
1263834822.423 0 10.10.122.241 TCP_MEM_HIT/200 657 GET
http://wpad/wpad.dat - NONE/- application/x-ns-proxy-autoconfig
I just double checked that (ForceType application/x-ns-proxy-autoconfig)
in my apache vhost config is working correctly. Also apache's mime.types
file is setup correctly for this particular item.
fixed the file which I also noticed has several other issues answering
my other other issue, my is 95% of my data being caught in the catch all
refresh_pattern instead of the mime type ones.
Point 2: Squid does not accept mime types in the refresh_pattern
directive.
This explains a few things.
Are you _sure_ that:
* the PAC file is not cached with old headers from before your changes?
Yes
* the PAC file is actually being fetched from the web server you are
expecting?
Yes
* this is an official build of Squid?
Yes, see below.
* nobody has applied third-party patches to it?
(none of the official Gentoo patches change mime.types.
http://sources.gentoo.org/viewcvs.py/gentoo-x86/net-proxy/squid/files/)
Fairly sure.
What headers does this produce when run on the Squid box?
squidclient -v -h wpad -p 80 /wpad.dat
I'm posting version and configuration at the bottom of this email.
Refresh patterns will be changed after this email is sent. This is a
standard gentoo install with the epoll USE flag.
[ebuild R ] net-proxy/squid-3.0.19 USE="caps epoll ldap mysql pam
samba sqlite ssl -icap-client (-ipf-transparent) -kerberos -kqueue
-logrotate* -nis (-pf-transparent) -postgres -radius -sasl (-selinux)
-snmp -zero-penalty-hit" 0 kB
(squidclient -v -h wpad -p 80 /wpad.dat) yeilds
headers: 'GET /wpad.dat HTTP/1.0
Accept: */*
'
HTTP/1.1 404 Not Found
Date: Tue, 19 Jan 2010 03:27:19 GMT
Server: Apache
Content-Length: 265
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /wpad.dat was not found on this server.</p>
<hr>
<address>Apache Server at localhost Port 80</address>
</body></html>
So I used GET instead.
(GET http://wpad/wpad.dat -USed)
GET http://wpad/wpad.dat
User-Agent: lwp-request/5.827 libwww-perl/5.831
GET http://wpad/wpad.dat --> 200 OK
Connection: close
Date: Tue, 19 Jan 2010 03:28:59 GMT
Accept-Ranges: bytes
Age: 412
ETag: "736a9e-119-47d6be3f06d80"
Server: Apache
Content-Length: 281
Content-Type: application/x-ns-proxy-autoconfig
Last-Modified: Mon, 18 Jan 2010 08:10:46 GMT
Client-Date: Tue, 19 Jan 2010 03:28:59 GMT
Client-Peer: 10.10.122.250:80
Client-Response-Num: 1
Of note for other Gentoo& Debian users: From mime.types # This file is
part of the app-misc/mime-types package, which is based on debian's
"mime-support".
So my question is now; how do I force squid to use the mime-type
delivered by the remote webserver without killing mime.types and thus
breaking my system in new and unexpected ways?
The official releases of Squid pass content-type headers through
unchanged. Something is broken.
On 1/15/2010 8:22 PM, Amos Jeffries wrote:
Jason Spegal wrote:
Is mime.conf what is used by refresh_pattern when mime types are used
for the regex?
No.
refresh_pattern uses a text regex against the requested URL string.
mime.conf is used by FTP and Gopher directory display to show the
icons.
Amos
Squid Cache: Version 3.0.STABLE19
configure options: '--prefix=/usr' '--build=i686-pc-linux-gnu'
'--host=i686-pc-linux-gnu' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc'
'--localstatedir=/var/lib' '--sysconfdir=/etc/squid'
'--libexecdir=/usr/libexec/squid' '--localstatedir=/var'
'--datadir=/usr/share/squid' '--with-default-user=squid'
'--enable-auth=basic,digest,negotiate,ntlm'
'--enable-removal-policies=lru,heap'
'--enable-digest-auth-helpers=password'
'--enable-basic-auth-helpers=DB,PAM,LDAP,SMB,multi-domain-NTLM,getpwnam,NCSA,MSNT'
'--enable-external-acl-helpers=ldap_group,wbinfo_group,ip_user,session,unix_group'
'--enable-ntlm-auth-helpers=SMB,fakeauth'
'--enable-negotiate-auth-helpers=' '--enable-useragent-log'
'--enable-cache-digests' '--enable-delay-pools' '--enable-referer-log'
'--enable-arp-acl' '--with-large-files' '--with-filedescriptors=8192'
'--enable-caps' '--disable-snmp' '--enable-ssl' '--disable-icap-client'
'--enable-http-violations' '--with-pthreads' '--with-aio'
'--enable-storeio=ufs,diskd,aufs,null' '--enable-linux-netfilter'
'--enable-epoll' 'build_alias=i686-pc-linux-gnu'
'host_alias=i686-pc-linux-gnu' 'CC=i686-pc-linux-gnu-gcc'
'CFLAGS=-march=pentium4m -O2 -pipe -fomit-frame-pointer'
'LDFLAGS=-Wl,-O1' 'CXXFLAGS=-march=pentium4m -O2 -pipe -fomit-frame-pointer'
From squid.conf:
acl allowed_sites url_regex
"/etc/squid/filtered_sites/allow/exceptions.allow"
acl blocked_sites url_regex "/etc/squid/filtered_sites/deny/ads.blocked"
acl blocked_sites url_regex "/etc/squid/filtered_sites/deny/porn.blocked"
acl blocked_sites url_regex "/etc/squid/filtered_sites/deny/spyware.blocked"
http_access allow allowed_sites
http_access deny blocked_sites
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.10.0.0/12
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access allow manager localnet
http_access deny manager all
http_access allow purge localhost
http_access allow purge localnet
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access deny all
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i
\.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay
\.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback
\.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i
\.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay
\.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl videocache_allow_url url_regex -i
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i
proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i
[a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i
bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i
va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i
\.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i
media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i
\.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com
.cdn.dailymotion.com
acl videocache_deny_url url_regex -i
http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
icp_access allow localnet
icp_access deny all
htcp_access allow localnet
htcp_access deny all
http_port 10.10.122.250:3128 transparent
http_port 10.10.123.1:3128 transparent
hierarchy_stoplist cgi-bin ?
cache_mem 1024 MB
maximum_object_size_in_memory 8192 KB
memory_replacement_policy heap LRU
cache_replacement_policy heap LFUDA
cache_dir aufs /mnt/archive3/squid_cache 65536 64 256
store_dir_select_algorithm round-robin
max_open_disk_fds 0
minimum_object_size 0 KB
maximum_object_size 8192000 KB
cache_swap_low 97
cache_swap_high 99
logformat squid %ts.%03tu %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
logformat squidmime %ts.%03tu %6tr %>a %Ss/%03Hs %<st %rm %ru %un
%Sh/%<A %mt [%>h] [%<h]
logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st %Ss:%Sh
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st
"%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
logformat superlog %tg.%03tu %14>a %-15>A %14<A %7tr %5rm HTTP/%3rv
%-4Hs %-23Ss %8Sh %10<st %43mt URL::%'-130ru RESPONSE HEADER::%"<h
access_log /var/log/squid/access.log squid
access_log /var/www/websites/admin/logs/squid_superlog.txt superlog
logfile_rotate 24
strip_query_terms on
buffered_logs on
acl dynamic_content urlpath_regex -i
\.(asp|aspx|php|pl|xml|rss|kml|cgi|py|pyc) #(\?.*)?$
acl dynamic_content urlpath_regex -i http://audio*pandora.com/*.mp*
acl dynamic_content urlpath_regex -i cgi-bin
cache deny dynamic_content
cache allow all
refresh_pattern -i kh*.google.com/? 43200 80% 259200 ignore-no-cache
ignore-private ignore-no-store ignore-auth override-expire
override-lastmod ignore-reload
refresh_pattern -i virtualearth.net/? 43200 80% 259200 ignore-no-cache
ignore-private ignore-no-store ignore-auth override-expire
override-lastmod ignore-reload
refresh_pattern application/* 43200 80% 259200 ignore-no-cache
ignore-private ignore-no-store ignore-auth
refresh_pattern audio/* 43200 80% 259200 ignore-no-cache ignore-private
ignore-no-store ignore-auth
refresh_pattern images/* 10080 16% 259200 ignore-no-cache ignore-private
ignore-no-store ignore-auth override-expire override-lastmod
refresh_pattern text/* 0 16% 259200 refresh-ims
refresh_pattern video/* 43200 80% 259200 ignore-no-cache ignore-private
ignore-no-store ignore-auth
refresh_pattern . 0 80% 259200 ignore-no-cache ignore-private
ignore-no-store ignore-auth
quick_abort_min -1 KB
quick_abort_max 16384000 KB
quick_abort_pct 5
read_ahead_gap 7 KB
negative_ttl 60 seconds
positive_dns_ttl 1 hours
negative_dns_ttl 5 minutes
range_offset_limit -1 KB
minimum_expiry_time 60 seconds
vary_ignore_expire on
reply_header_access Pragma deny all
reply_header_access Cache-Control deny all
shutdown_lifetime 3 seconds
cache_mgr jspegal
httpd_suppress_version_string on
visible_hostname ichiban
client_persistent_connections on
server_persistent_connections on
persistent_connection_after_error off
detect_broken_pconn on
icp_port 3130
log_icp_queries on
short_icon_urls on
deny_info about:blank blocked_sites
nonhierarchical_direct on
prefer_direct off
check_hostnames on
dns_retransmit_interval 1 seconds
dns_timeout 1 minutes
dns_nameservers 10.10.122.250
hosts_file /etc/hosts
ipcache_size 12000
ipcache_low 90
ipcache_high 95
fqdncache_size 12000
memory_pools on
memory_pools_limit 256 MB
forwarded_for off
cachemgr_passwd none all
client_db on
refresh_all_ims off
maximum_single_addr_tries 3
retry_on_error on
offline_mode off
coredump_dir /var/cache/squid
balance_on_multiple_ip off
pipeline_prefetch on
Hopefully this is helpful.
