On 09/12/2013 03:12 PM, Pavel Kazlenka wrote:
Thank you Amos,
On 09/12/2013 02:53 PM, Amos Jeffries wrote:
On 12/09/2013 10:51 p.m., Pavel Kazlenka wrote:
Hi gentlemen,
I'm trying to cache youtube videos following
http://wiki.squid-cache.org/Features/StoreID guide.
But seems like squid rejects to cache the content because original
server returns header 'Cache-control:private' and 'refresh pattern
... ignore-private' doesn't take effect. Here is debug log entries
that are interesting here (I think):
2013/09/11 19:24:24.842 kid1| helper.cc(419) helperSubmit:
buf[975]=http://r16---sn-4g57ln7e.c.youtube.com/videoplayback?ratebypass=yes&itag=43&ip=212.98.189.159&key=yt1&upn=6KCvBVLs-yM&mt=1378980214&fexp=919118%2C924606%2C929117%2C929121%2C929906%2C929907%2C929922%2C929127%2C929129%2C929131%2C929930%2C936403%2C92
5726%2C936310%2C925720%2C925722%2C925718%2C925714%2C929917%2C906945%2C929933%2C920302%2C906842%2C913428%2C920605%2C919811%2C935020%2C935021%2C913563%2C919373%
2C930803%2C908536%2C932211%2C938701%2C931924%2C934005%2C936308%2C909549%2C900816%2C912711%2C904494%2C904497%2C939903%2C900375%2C900382%2C934507%2C907231%2C936
312%2C906001&id=2c5e89c0af6c8804&expire=1379004381&sver=3&ipbits=8&cp=U0hWTlBLUl9NS0NONl9IRVZEOjNVQko4ZGFaMGcz&ms=au&source=youtube&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&mv=m&cpn=Ph7LcSRYt1STlsoQ&signature=BB797D0EFC5182670EF89E95EFBB6E5D12F49B8F.6404F37419A96004F8DDCA2CAB901101A
30082CA&ptk=youtube_none&pltype=contentugc 192.168.10.8/- - GET
myip=192.168.10.245 myport=3128
...
2013/09/11 19:24:24.843 kid1| helper.cc(919) helperHandleRead:
accumulated[77]=OK
store-id=http://video-srv.youtube.com.squid.internal/43&2c5e89c0af6c8804";
2013/09/11 19:24:24.843 kid1| store_dir.cc(786) get: none of 1
cache_dirs have 028E8844ECA93A634459175C8C0D463D
2013/09/11 19:24:24.843 kid1| store.cc(840) storeCreateEntry:
storeCreateEntry:
'http://video-srv.youtube.com.squid.internal/43&2c5e89c0af6c8804'
2013/09/11 19:24:25.032 kid1| http.cc(396) cacheableReply: NO
because server reply Cache-Control:private
My squid.conf:
acl rewritedoms dstdomain .c.youtube.com
store_id_program /home/tester/squid/libexec/storeid_file_rewrite
/home/tester/squid/db.txt
store_id_children 40 startup=10 idle=5 concurrency=0
store_id_access allow rewritedoms
store_id_access deny all
...
refresh_pattern ^http://video-srv\.youtube\.squid\.internal/.* 10080
80% 79900 override-lastmod override-expire ignore-reload
ignore-must-revalidate ignore-private
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
#refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
Squid version is squid-3.4.0.1-20130910-r12989.
So my questions are:
1) Am I right that refresh_pattern ... ignore-private should work
here and help to cache reply with CC:Private?
Yes the exact code is:
if ( Cache-Control is present AND contains "private" AND
ignore-private is *absent*)
display that "NO because server reply Cache-Control:private"
message etc.
2) Is there another (may be more correct) way to force squid to
cache replies with CC:Private?
There is no _correct_ way to abuse the RFC protocol standard. This
ignore-private explicitly forbidden behaviour for any cache shared by
more than 1 person.
It is officially only made available in Squid to allow single-person
installations to operate a proxy between multiple devices.
Sure. I meant correct from squid's internals point of view (may be
dedicated directive).
3) How can I check that 'refresh_pattern' config string is parsed
correctly and all options are remembered by squid?
To check it is parsed correctly use:
squid -k parse
To check what the running config is you can produce a config file
dump using the cache manager interface:
* ensure that you have a cachemgr_passwd (or "none") explicitly
defined for either the "config" or the "all" reports.
http://www.squid-cache.org/Doc/config/cachemgr_passwd/
* ensure that your http_access rules involving "manager" ACL permit
you access to the proxy management interface.
* fetch http://example.com/squid-internal-mgr/config in your browser
The result should be a TXT format listing of all the squid.conf
settings (including the defaults) which Squid is using.
My config obtained in this way includes 'refresh_pattern
^http://video-srv\.youtube\.squid\.internal/.* 10080 80% 79900
override-expire override-lastmod ignore-reload ignore-must-revalidate
ignore-private', so I guess that this is definitely a bug. I'm going
to open new defect on bugzilla. Will see if this will be fixed in
3.4.0.2 when 'unknown_cfg_function' patch is accepted. Any objections?
Sorry, I checked with 'refresh_pattern ^.* 10080 80% 79900
override-expire override-lastmod ignore-reload ignore-must-revalidate
ignore-private' and file considered cacheable. So problem is in
incorrect regexp, I guess.
Amos
