Re: Re: Cache url's with '?' question marks

Adrian Chadd <adrian@xxxxxxxxxxxxxxx> · Sun, 16 Mar 2008 19:15:43 +0900

Erk, HTML email, thats not even done right! *smacks your client* :)

Uhm, start by looking at the header logging feature (mime_${SOMETHING} in squid.conf) to log
request and reply headers.

I'd then use those client requests to create test cases to feed to Squid.

Adrian

On Fri, Mar 14, 2008, Saul Waizer wrote:
> <HTML>
> <P>Adrian,</P>
> <P>QUERY ACL has been removed for over a week, can you give me some pointers as far as looking at the MISSes</P>
> <P>Thanks!<br>
> 
> <br>
> 
> <br>
> 
> <br>
> 
> <B>On Fri , Adrian Chadd <ADRIAN@xxxxxxxxxxxxxxx>sent:<br>
> 
> </P></B>
> <BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #f5f5f5 2px solid; MARGIN-RIGHT: 0px">Caching dynamic content doesn't work "like that".<br>
> 
> <br>
> 
> Firstly, removing the QUERY ACL gives you the ability to cache dynamic<br>
> 
> content that returned explicit lifetime.<br>
> 
> <br>
> 
> You need to look at all of those MISSes and see why Squid isn't caching<br>
> 
> them. Its hard to tell from where I'm sitting.<br>
> 
> <br>
> 
> <br>
> 
> <br>
> 
> <br>
> 
> Adrian<br>
> 
> <br>
> 
> <br>
> 
> On Fri, Mar 14, 2008, Saul Waizer wrote:<br>
> 
> <SPAN style="COLOR: #ff0000">&gt; -----BEGIN PGP SIGNED MESSAGE-----</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; Hash: SHA1</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; Amos,</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; I've implemented the example you sent on Dynamic Content but so far i</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; regret to say that no improvement has been made on the hit ratio</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; I added the following to my squid.conf</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; refresh_pattern (/cgi-bin/|\?) 0 0% 0</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; refresh_pattern . 0 20% 4320</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; acl mydomain dstdomain .mydomain.com</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; cache allow mydomain</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; my stats look something like this:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; 67.5103% TCP_MISS/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; 6.07349% TCP_HIT/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; 4.55681% TCP_MEM_HIT/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; 1.59761% TCP_IMS_HIT/304</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; Any help is appreciated.</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; Thanks</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; Amos Jeffries wrote:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; Adrian Chadd wrote:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt; G'day,</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt;</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt; Just remove the QUERY ACL and the cache ACL line using "QUERY" in it.</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt; Then turn on header logging (log_mime_hdrs on) and see if the replies</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt; to the dynamically generated content is actually giving caching info.</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt;</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt;</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt;</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt; Adrian</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; <A href="http://mail.hoodiny.com/parse.php?redirect=http%3A%2F%2Fwiki.squid-cache.org%2FConfigExamples%2FDynamicContent"; target=_blank><SPAN style="COLOR: #ff0000">http://wiki.squid-cache.org/ConfigExamples/DynamicContent</SPAN></A></SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; Amos</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt;</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt; On Fri, Feb 29, 2008, Saul Waizer wrote:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; Hello List,</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; I am having problems trying to cache images*/content that comes from a</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; URL containing a question mark on it ('?')</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; Background:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; I am running squid Version 2.6.STABLE17 on FreeBSD 6.2 as a reverse</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; proxy to accelerate content hosted in America served in Europe.</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; The content comes from an application that uses TOMCAT so a URL</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; requesting dynamic content would look similar to this:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; <A href="http://mail.hoodiny.com/parse.php?redirect=http%3A%2F%2Fdomain.com%2Fstorage%2Fstorage%3FfileName%3D%2F.domain.com-1%2Fusr%2F14348%2Fimage%2Fthumbnail%2Fth_8837728e67eb9cce6fa074df7619cd0d193_1_.jpg"; target=_blank><SPAN style="COLOR: #ff0000">http://domain.com/storage/storage?fileName=/.domain.com-1/usr/14348/image/thumbnail/th_8837728e67eb9cce6fa074df7619cd0d193_1_.jpg</SPAN></A></SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; The result of such request always results on a MISS with a log similar</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; to this:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; TCP_MISS/200 8728 GET <A href="http://mail.hoodiny.com/parse.php?redirect=http%3A%2F%2Fdomain.com%2Fstorage%2Fstorage%3F"; target=_blank><SPAN style="COLOR: #ff0000">http://domain.com/storage/storage?</SPAN></A> -</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; FIRST_UP_PARENT/server_1 image/jpg</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; I've added this to my config: acl QUERY urlpath_regex cgi-bin as you can</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; see bellow but it makes no difference and I tried adding this:</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl QUERY urlpath_regex cgi-bin \? and for some reason ALL requests</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; result in a MISS.</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; Any help is greatly appreciated.</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; My squid config looks like this: (obviously real ip's were changed)</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; ################# STANDARD ACL'S ###############################</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl all src 0.0.0.0/0.0.0.0</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl manager proto cache_object</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl localhost src 127.0.0.1/255.255.255.255</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl to_localhost dst 127.0.0.0/8</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; ################# REVERSE CONFIG FOR SITE #####</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; http_port 80 accel vhost</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_peer 1.1.1.1 parent 80 0 no-query originserver name=server_1</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl sites_server_1 dstdomain domain.com</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; ################# REVERSE ACL'S FOR OUR DOMAINS ##############</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl ourdomain0 dstdomain <A href="http://mail.hoodiny.com/parse.php?redirect=http://www.domain.com"; target=_blank>www.domain.com</A></SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl ourdomain1 dstdomain domain.com</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; http_access allow ourdomain0</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; http_access allow ourdomain1</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; http_access deny all</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; icp_access allow all</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; ################ HEADER CONTROL ###############</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; visible_hostname cacheA.domain.com</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_effective_user nobody</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; forwarded_for on</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; follow_x_forwarded_for allow all</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; header_access All allow all</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; ############### SNMP CONTROL ###############</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; snmp_port 161</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl snmppublic snmp_community public1</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; snmp_access allow all</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; ############## CACHE CONTROL ################</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; access_log /usr/local/squid/var/logs/access.log squid</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; acl QUERY urlpath_regex cgi-bin</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_mem 1280 MB</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_swap_low 95</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_swap_high 98</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; maximum_object_size 6144 KB</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; minimum_object_size 1 KB</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; maximum_object_size_in_memory 4096 KB</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_dir ufs /storage/ram_dir1 128 16 256</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_dir ufs /storage/cache_dir1 5120 16 256</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_dir ufs /storage/cache_dir2 5120 16 256</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; cache_dir ufs /storage/cache_dir3 5120 16 256</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; Also here is the result of a custom script I made to parse the</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; access.log that will sort and display the top 22 responses so I can</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; compare them with cacti, I am trying to increase the Hit ratio but so</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; far is extremely low.</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 1 571121 69.3643% TCP_MISS/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 2 98432 11.9549% TCP_HIT/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 3 51590 6.26576% TCP_MEM_HIT/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 4 47009 5.70938% TCP_MISS/304</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 5 17757 2.15664% TCP_IMS_HIT/304</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 6 11982 1.45525% TCP_REFRESH_HIT/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 7 11801 1.43327% TCP_MISS/404</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 8 6810 0.827095% TCP_MISS/500</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 9 2508 0.304604% TCP_MISS/000</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 10 1323 0.160682% TCP_MISS/301</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 11 1151 0.139792% TCP_MISS/403</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 12 1051 0.127647% TCP_REFRESH_HIT/304</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 13 430 0.0522248% TCP_REFRESH_MISS/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 14 127 0.0154245% TCP_CLIENT_REFRESH_MISS/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 15 83 0.0100806% TCP_MISS/401</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 16 81 0.00983769% TCP_CLIENT_REFRESH_MISS/304</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 17 35 0.00425085% TCP_MISS/503</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 18 20 0.00242906% TCP_DENIED/400</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 19 19 0.00230761% TCP_HIT/000</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 20 19 0.00230761% TCP_DENIED/403</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 21 14 0.00170034% TCP_SWAPFAIL_MISS/200</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; 22 1 0.000121453% TCP_SWAPFAIL_MISS/30</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; Thanks!</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; &gt;&gt;</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; -----BEGIN PGP SIGNATURE-----</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; Version: GnuPG v1.4.6 (GNU/Linux)</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; Comment: Using GnuPG with Mozilla - <A href="http://mail.hoodiny.com/parse.php?redirect=http%3A%2F%2Fenigmail.mozdev.org"; target=_blank><SPAN style="COLOR: #ff0000">http://enigmail.mozdev.org</SPAN></A></SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; </SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; iD8DBQFH2ugEAcr37anguZsRAixSAJ9GioRmL42D1bOSVveMKYcpi21fygCfd4VP</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; UMqi1CB3vQ5IeaTblK/vYQM=</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; =6XUg</SPAN><br>
> 
> <SPAN style="COLOR: #ff0000">&gt; -----END PGP SIGNATURE-----</SPAN><br>
> 
> <br>
> 
> -- <br>
> 
> - Xenion - <A href="parse.php?redirect=http%3A%2F%2Fwww.xenion.com.au%2F" target=_blank><SPAN style="COLOR: #ff0000">http://www.xenion.com.au/</SPAN></A> - VPS Hosting - Commercial Squid Support -<br>
> 
> - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -<br>
> 
> <br>
> 
> </BLOCKQUOTE><BR></HTML>

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -