Cached Contents are so many .... -Youtube -GoogleSyndication -ADS and it will always give me a strong headache to follow every ADby caching its content not to block it because blocking any ADS URL would present a javascript error at the browser's side!. -VideoZer - android Google ( MARKET .. etc ) - Ziddu - xHamster - SoundCloud - Some websites that have a cdn folder which generates a key or long key in the middle of the URL which everytime you refresh the website then it will generate a dynamic folder incase if it has an extension jpg or any multimedia extension and sometimes you see URL which has an end like that abc.jpg;blablabla or abc.jpg?blablabla so I try to remove that CDN Folder and at the same time i also remove everything which comes after the extension jpg if has ";" or "?" . - wlxrs.com - reverbnation , megavideo , xaxtube - NOGOMI/ AHLASOOT .. for example to save bandwidth !!! . usually before you download the MP3 file they allow you to listen online through their web player which has the same file size and same file_name but if you download the file then you get a different domain name which made me think just to rewrite the URL which matches and save the same URL into cache if a client wanted to listen or download and also alot of websites has the same idea. - vkontakte , depositfiles , eporner , 4shared , letitbit , sendspace , filesonic , uploaded , uploading , turbobit, wupload , redtubefiles , filehippo , oron , rapishare , tube8 , pornhub , xvideos , telstra , scribdassets , przeklej , hardsextube , fucktube , imageshack , beeg , yahoo videos , youjizz , gcdn for example , look at this URL : #http://b2htks6oia9cm5m4vthd6hhulo.gcdn.biz/d/r/*/FileName WIthout Extension (*) DYNAMIC CONTENT and just look into the sub domain how its like! and so many websites but unfotunately EASY-SHARE is using POST response and I cant cache it. Lately I was monitoring NOKIA Phones and I added OVI STORE. Do you have any website that its not cacheable or using CDN or something because Im really interested to look at :) . Ghassan On Mon, Apr 30, 2012 at 2:53 AM, Eliezer Croitoru <eliezer@xxxxxxxxxxxx> wrote: > On 30/04/2012 02:18, Ghassan Gharabli wrote: >> >> Hello Eliezer, >> >> Are you trying to save all video chunks into same parts or capture / >> download the whole video object through CURL or whatever! but i dont >> think it should work since it will occure an error with the new >> Youtube Player. >> >> What I have reached lately is saving same youtube video chunks that >> has youtube 360p itag=34 ,480p itag=35 without saving its itag since >> i want to save more bandwidth (thats why i only wrote scripts to you >> as an example) which means if someone wants to watch 480p then he >> would get the cached 360p contents thats why i didnt add the itag but >> if he thought of watching 720p and above then another script would >> catch it matching ITAG 37 , 22 ... I know that is not the best >> solution but at least its working pretty well with no erros at all as >> long as the client can always fast forward . >> >> Im using Squid 2.7 Stable9 compiled on windows 64-bit with PERL x64. >> >> Regarding the 302 Redirection .. I have made sure to update the >> source file client_side.c to fix the loop 302 Redirection but really I >> dont have to worry now about anything so what is your target regarding >> Youtube with argument Range and whats the problem till now ? >> >> I have RAID with 5 HDD and the average HTTP Requests per minute : >> 2732.6 and because I want to save more bandwidth I try to analyze HTTP >> Requests so i can always update my perl script to match most wanted >> websites targetting Videos , Mp3 etc. >> >> For a second I thought of maybe someone would help to compile an >> intelligent external helper script that would capture the whole >> byte-range and I know it is really hard to do that since we are >> dealing with byte-range. >> >> I only have one question that is always teasing me .. what are the >> comnparison between SQUID and BLUE COAT so is it because it is a >> hardware perfromance or just it has more tricks to cache everything >> and reach a maximum ratio ? >> >> >> >> Ghassan > > i was messing with store_url_rewrite and url_rewrite quite some time just > for knowledge. > > i was researching every concept exists with squid until now. > a while back (year or more) i wrote store_url_rewrite using java and posted > the code somewhere. > the reason i was using java was because it's the fastest and simples from > all other languages i know (ruby perl python). > i was saving bandwidth using nginx because it was simple to setup. > i dont really like the idea of faking my users about the quality and also it > can make a very problematic state that the user can get partial HQ content > what will break the user from watching videos. > > i dont really have any problems with the ranges. > i just noticed that there are providers in my country that are not using the > "range" parameter but the "begin" parameter... > > i will be very happy to get the client_side.c patch to fix the 302 loop. > > the problem with external_helper is that it's not really needed. > if you need something you can use ICAP for that but you still there is a lot > of work to be dont so as far it seems that store_url_rewrite is the best > option. > > BLUECOAT has option to relate objects also using ETAG and other objects > parameters so it makes it a very robust caching system. > > nginx does the work just fine for youtube videos but it seems like some > headers are messing and should be added. > > by the way, > What video\mp3 sites are you caching using your scripts? > > > Eliezer > > >> >> >> >> On Mon, Apr 30, 2012 at 1:29 AM, Eliezer Croitoru<eliezer@xxxxxxxxxxxx> >> wrote: >>> >>> On 24/04/2012 21:02, Eliezer Croitoru wrote: >>>> >>>> >>>> as for some people asking me recently about youtube cache i have checked >>>> again and found that youtube changed their video uris and added an >>>> argument called "range" that is managed by the youtube player. >>>> the original url\uri dosnt include range but the youtube player is using >>>> this argument to save bandwidth. >>>> >>>> i can implement the cahing with ranges on nginx but i dont know yet the >>>> way that range works. >>>> it can be based on user bandwidth or "fixed" size of chunkes. >>>> >>>> if someone up to the mission of analyzing it a bit more to understand it >>>> so the "range" cache will be implemented i will be happy to get some >>>> help with it. >>>> >>>> Thanks, >>>> Eliezer >>>> >>>> >>> as for now the "minimum_object_size 512 bytes" wont do the trick for 302 >>> redirection on squid2.7 because the 302 response is 963 big size. >>> so i have used: >>> minimum_object_size 1024 bytes >>> just to make sure it will work. >>> and also this is a youtube videos dedicated server so it's on with this >>> limit. >>> >>> Regards, >>> >>> Eliezer >>> >>> -- >>> Eliezer Croitoru >>> https://www1.ngtech.co.il >>> IT consulting for Nonprofit organizations >>> eliezer<at> ngtech.co.il > > > > -- > Eliezer Croitoru > https://www1.ngtech.co.il > IT consulting for Nonprofit organizations > eliezer <at> ngtech.co.il
