Now I am encountering an issue: I worked on the ICAP service and it works fine to find and extract the unique identifier for the googlevideo id prediction. Now I have a working ICAP+StoreID helpers but from an unknown reason it seems that squid find's a match in the cache but still serves the content from the origin server. What might be causing it? I tried to run a basic debug but I need some help in the lookup, what to look for? What might be causing it? I am attaching "debug_options ALL,1 11,3" output of a request(I am removing the Accept-Enconding from the request, which doesn't seems to affect anything) Also I am attaching store.log output which shows a release of the object just before it's being swaped out. What debug_options might help in this scenario? I am kind of clueless since I have not touched the subject for a very long time. ## cache.log 2017/05/17 05:00:23.861 kid1| 11,3| http.cc(1061) persistentConnStatus: local=192.168.89.12:36708 remote=194.90.196.53:443 FD 37 flags=1 eof=0 2017/05/17 05:00:24.351 kid1| 11,2| client_side.cc(2364) parseHttpRequest: HTTP Client local=192.168.89.12:13128 remote=192.168.89.58:61588 FD 89 flags=1 2017/05/17 05:00:24.351 kid1| 11,2| client_side.cc(2365) parseHttpRequest: HTTP Client REQUEST: --------- GET /videoplayback?sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495008016&ipbits=0&requiressl=yes&keepalive=yes&mt=1494986305&upn=gDVDWK1Q9qM&mime=video%2Fmp4&initcwndbps=892500&pl=20&ei=sK4bWZ6AHMGsW7bJgcgP&clen=68830605&itag=135&gir=yes&mm=31&mn=sn-aigllndd&dur=3835.833&id=o-AOxPmibb2YStmY_Tqj-VR0X40QeAgJtRsH3sB1Ma2WO1&key=yt6&ip=84.95.212.160&lmt=1428485111698865&source=youtube&mv=m&ms=au&alr=yes&ratebypass=yes&signature=C6DF66F0AC390B367375FF487E7FC11CA01BC834.9392D0F984CBCAE83A1FB380BE826C93E8739747&cpn=NowGUcZPl2y4ArWs&c=WEB&cver=1.20170511&range=1029733-2546112&rn=9&rbuf=34495 HTTP/1.1 Host: r15---sn-aigllndd.googlevideo.com User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0 Accept: */* Accept-Language: en-US,he;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate, br Referer: https://www.youtube.com/ Origin: https://www.youtube.com Connection: keep-alive ---------- 2017/05/17 05:00:24.358 kid1| 11,3| http.cc(2274) httpStart: GET http://ytgv.squid.internal/id=X727AYHiNLM&itag=135&range=1029733-2546112 2017/05/17 05:00:24.358 kid1| 11,2| http.cc(2230) sendRequest: HTTP Server local=192.168.89.12:54702 remote=209.85.230.84:443 FD 90 flags=1 2017/05/17 05:00:24.358 kid1| 11,2| http.cc(2231) sendRequest: HTTP Server REQUEST: --------- GET /videoplayback?sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495008016&ipbits=0&requiressl=yes&keepalive=yes&mt=1494986305&upn=gDVDWK1Q9qM&mime=video%2Fmp4&initcwndbps=892500&pl=20&ei=sK4bWZ6AHMGsW7bJgcgP&clen=68830605&itag=135&gir=yes&mm=31&mn=sn-aigllndd&dur=3835.833&id=o-AOxPmibb2YStmY_Tqj-VR0X40QeAgJtRsH3sB1Ma2WO1&key=yt6&ip=84.95.212.160&lmt=1428485111698865&source=youtube&mv=m&ms=au&alr=yes&ratebypass=yes&signature=C6DF66F0AC390B367375FF487E7FC11CA01BC834.9392D0F984CBCAE83A1FB380BE826C93E8739747&cpn=NowGUcZPl2y4ArWs&c=WEB&cver=1.20170511&range=1029733-2546112&rn=9&rbuf=34495 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0 Accept: */* Accept-Language: en-US,he;q=0.7,en;q=0.3 Referer: https://www.youtube.com/ Origin: https://www.youtube.com Host: r15---sn-aigllndd.googlevideo.com Cache-Control: max-age=4794000 Connection: keep-alive ---------- 2017/05/17 05:00:24.436 kid1| ctx: enter level 0: 'http://ytgv.squid.internal/id=X727AYHiNLM&itag=135&range=1029733-2546112' 2017/05/17 05:00:24.436 kid1| 11,3| http.cc(694) processReplyHeader: processReplyHeader: key 'C40AE216D4CE802B5FAE71A34B75A843' 2017/05/17 05:00:24.436 kid1| 11,2| http.cc(735) processReplyHeader: HTTP Server local=192.168.89.12:54702 remote=209.85.230.84:443 FD 90 flags=1 2017/05/17 05:00:24.436 kid1| 11,2| http.cc(736) processReplyHeader: HTTP Server REPLY: --------- HTTP/1.1 200 OK Last-Modified: Wed, 08 Apr 2015 09:25:11 GMT Content-Type: video/mp4 Date: Wed, 17 May 2017 02:00:24 GMT Expires: Wed, 17 May 2017 02:00:24 GMT Cache-Control: private, max-age=21292 Accept-Ranges: bytes Content-Length: 1516380 Connection: keep-alive Alt-Svc: quic=":443"; ma=2592000 Access-Control-Allow-Origin: https://www.youtube.com Access-Control-Allow-Credentials: true Timing-Allow-Origin: https://www.youtube.com Access-Control-Expose-Headers: Client-Protocol, Content-Length, Content-Type, X-Bandwidth-Est, X-Bandwidth-Est2, X-Bandwidth-App-Limited, X-Bandwidth-Est-App-Limited, X-Bandwidth-Est-Comp, X-Bandwidth-Avg, X-Head-Time-Sec, X-Head-Seqnum, X-Sequence-Num, X-Segment-Lmt, X-Walltime-Ms X-Content-Type-Options: nosniff Server: gvs 1.0 ---------- 2017/05/17 05:00:24.436 kid1| ctx: exit level 0 2017/05/17 05:00:24.436 kid1| 11,3| Client.cc(936) noteAdaptationAclCheckDone: no adapation needed 2017/05/17 05:00:24.436 kid1| ctx: enter level 0: 'http://ytgv.squid.internal/id=X727AYHiNLM&itag=135&range=1029733-2546112' 2017/05/17 05:00:24.436 kid1| 11,3| http.cc(905) haveParsedReplyHeaders: HTTP CODE: 200 2017/05/17 05:00:24.437 kid1| ctx: exit level 0 2017/05/17 05:00:24.437 kid1| 11,2| client_side.cc(1408) sendStartOfMessage: HTTP Client local=192.168.89.12:13128 remote=192.168.89.58:61588 FD 89 flags=1 2017/05/17 05:00:24.437 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client REPLY: --------- HTTP/1.1 200 OK Last-Modified: Wed, 08 Apr 2015 09:25:11 GMT Content-Type: video/mp4 Date: Wed, 17 May 2017 02:00:24 GMT Expires: Wed, 17 May 2017 02:00:24 GMT Cache-Control: private, max-age=21292 Accept-Ranges: bytes Content-Length: 1516380 Alt-Svc: quic=":443"; ma=2592000 Access-Control-Allow-Origin: https://www.youtube.com Access-Control-Allow-Credentials: true Timing-Allow-Origin: https://www.youtube.com Access-Control-Expose-Headers: Client-Protocol, Content-Length, Content-Type, X-Bandwidth-Est, X-Bandwidth-Est2, X-Bandwidth-App-Limited, X-Bandwidth-Est-App-Limited, X-Bandwidth-Est-Comp, X-Bandwidth-Avg, X-Head-Time-Sec, X-Head-Seqnum, X-Sequence-Num, X-Segment-Lmt, X-Walltime-Ms X-Content-Type-Options: nosniff Server: gvs 1.0 X-Cache: MISS from filter X-Cache-Lookup: HIT from filter:3128 Connection: keep-alive ---------- ## End of cache.log ## store.log 1494988764.096 RELEASE 00 000000FC 9F3457302C3C18CC9420F78989501F21 ? ? ? ? ?/? ?/? ? ? 1494988764.180 SWAPOUT 00 000001C8 9F3457302C3C18CC9420F78989501F21 200 1494988764 1425925658 1495009739 audio/webm 414553/414553 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=76689610&source=youtube&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495010039&dur=3836.501&ei=l7YbWfjyEdPacseFsOgB&initcwndbps=847500&mt=1494988343&requiressl=yes&ip=84.95.212.160&ms=au&gir=yes&mv=m&pl=20&mime=audio%2Fwebm&id=o-AAyoF-88PGHEqhkC-hBvKEPjiIeUoDi6DohmCQ0WoGs8&mn=sn-aigllndd&mm=31&ipbits=0&itag=251&keepalive=yes&lmt=1425925658208642&key=yt6&upn=JGXkLnd5jwI&alr=yes&ratebypass=yes&signature=7A20E093615AC902AAE33D726AD3F9BE3934DFDD.BDCC662814E40739201EA9EE69D1058F7C74C7A1&cpn=ZWjhgUtxrr_va9Vr&c=WEB&cver=1.20170511&range=8837089-9251641&rn=32&rbuf=352381 1494988783.284 RELEASE 00 000000FD 0436BF4E1A398F3365103BE514907CD5 ? ? ? ? ?/? ?/? ? ? 1494988783.366 SWAPOUT 00 000001C9 0436BF4E1A398F3365103BE514907CD5 200 1494988783 1425925658 1495009739 audio/webm 413087/413087 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=76689610&source=youtube&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495010039&dur=3836.501&ei=l7YbWfjyEdPacseFsOgB&initcwndbps=847500&mt=1494988343&requiressl=yes&ip=84.95.212.160&ms=au&gir=yes&mv=m&pl=20&mime=audio%2Fwebm&id=o-AAyoF-88PGHEqhkC-hBvKEPjiIeUoDi6DohmCQ0WoGs8&mn=sn-aigllndd&mm=31&ipbits=0&itag=251&keepalive=yes&lmt=1425925658208642&key=yt6&upn=JGXkLnd5jwI&alr=yes&ratebypass=yes&signature=7A20E093615AC902AAE33D726AD3F9BE3934DFDD.BDCC662814E40739201EA9EE69D1058F7C74C7A1&cpn=ZWjhgUtxrr_va9Vr&c=WEB&cver=1.20170511&range=9251642-9664728&rn=33&rbuf=372381 1494988939.087 RELEASE 00 000001EB 2257A59BCD5457AC526432A31D8C1DAE ? ? ? ? ?/? ?/? ? ? 1494988939.176 SWAPOUT 00 00000215 2257A59BCD5457AC526432A31D8C1DAE 200 1494988939 1425925658 1495010229 audio/webm 290838/290838 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=76689610&upn=Ar8EsOEX6u8&source=youtube&expire=1495010529&lmt=1425925658208642&mime=audio%2Fwebm&gir=yes&requiressl=yes&keepalive=yes&id=o-AH3RVksarviuBTrz7-iVa695QDmwgtd25mI2pwtDvNDe&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&initcwndbps=822500&ms=au&itag=251&mt=1494988830&dur=3836.501&mv=m&pl=20&mm=31&ipbits=0&mn=sn-aigllndd&key=yt6&ei=gbgbWdf_FtW9cseknvgM&ip=84.95.212.160&alr=yes&ratebypass=yes&signature=7C1C7F0389FF6102FAAFB4F4978E8A2BEAE64D40.596DE1F2DEAD3091CD0B49354E49A42BEE812362&cpn=Xy6NBwdfsknjfruc&c=WEB&cver=1.20170511&range=327790-618627&rn=8&rbuf=16213 1494988940.164 RELEASE 00 00000207 E306D0FF544DCD9B997C189364B9173D ? ? ? ? ?/? ?/? ? ? 1494988940.493 SWAPOUT 00 0000021B E306D0FF544DCD9B997C189364B9173D 200 1494988940 1428485111 1495010229 video/mp4 1516380/1516380 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=68830605&upn=Ar8EsOEX6u8&source=youtube&expire=1495010529&lmt=1428485111698865&mime=video%2Fmp4&gir=yes&requiressl=yes&keepalive=yes&id=o-AH3RVksarviuBTrz7-iVa695QDmwgtd25mI2pwtDvNDe&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&initcwndbps=822500&ms=au&itag=135&mt=1494988830&dur=3835.833&mv=m&pl=20&mm=31&ipbits=0&mn=sn-aigllndd&key=yt6&ei=gbgbWdf_FtW9cseknvgM&ip=84.95.212.160&alr=yes&ratebypass=yes&signature=B38EC7D16A460518D56AB4AEC2686F8417E4F66D.5C0A937164F499F9951C0AC2ED4E3C1655718B98&cpn=Xy6NBwdfsknjfruc&c=WEB&cver=1.20170511&range=1029733-2546112&rn=10&rbuf=38883 ## End of store.log Thanks, Eliezer ---- Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: eliezer@xxxxxxxxxxxx -----Original Message----- From: squid-users [mailto:squid-users-bounces@xxxxxxxxxxxxxxxxxxxxx] On Behalf Of Eliezer Croitoru Sent: Tuesday, May 16, 2017 4:04 AM To: squid-users@xxxxxxxxxxxxxxxxxxxxx Subject: Experimental YouTube Caching helper\tool and SQUID 3.5.25 + 4.0.19 RPM's RELEASED Hey List, I have been working a while ago on a tool(ICAP service) that will do the next: - Snatch YouTube main video pages(watch?...) on the fly - Fetch them - Parse them - Find the key to predict the ID of the X.googlevideo.com/Y which will match the YouTube video - Store the key with the correlated video ID in a redis DB for an hour - Then a StoreID helper will analyze X.googlevideo.com/Y urls and will try to see if it finds the matching a YouTube video by the ID on the redis DB The concept is kind of "divide and conquer" while later if I will have time there will be an option to combine these two things into one ICAP service that will help to cache YouTube videos and probably also Windows Updates using squid. Currently the ICAP service code will not be published but the binaries and the resources which I have used to create the tool will be published. (Something like the ingredients without the recipe due to this task being considered a caching master level and not just any novice) However my email is wide open for questions requests and tips for anyone that wishes to get answers, directions and guidance. @Alex, What will encourage you to help with an improved patch to add ICAP X-StoreID (or else) response header capability that will help me to write a complete StoreID solution based on ICAP alone? @Others, if you are interested in this solution publication please respond to this thread and CC to me. Thanks, Eliezer * I have released Squid-Cache 3.5.25 and 4.0.19 RPMS but Life takes more of my time and as I get busy I didn't have enough time to publish the article like in the past. The article topic is "ENCRYPTION, HOW FAR WILL YOU GO?" at [http://www1.ngtech.co.il/wpe/?p=430] ---- Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: eliezer@xxxxxxxxxxxx _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
