well a nice list.
all the filesharing sites are not something i really want to cache
because the effort to understand most of the sites is too much then the
benefit.
megavideo got shutdown.
what i'm and others are very interested is microsoft windows updates.
they are using a 206 range request that make it's impossible to cache
with squid as far as i understand.
Eliezer
On 01/05/2012 13:41, Ghassan Gharabli wrote:
Cached Contents are so many ....
-Youtube
-GoogleSyndication
-ADS and it will always give me a strong headache to follow every ADby
caching its content not to block it because blocking any ADS URL would
present a javascript error at the browser's side!.
-VideoZer
- android Google ( MARKET .. etc )
- Ziddu
- xHamster
- SoundCloud
- Some websites that have a cdn folder which generates a key or long
key in the middle of the URL which everytime you refresh the website
then it will generate a dynamic folder incase if it has an extension
jpg or any multimedia extension and sometimes you see URL which has an
end like that abc.jpg;blablabla or abc.jpg?blablabla so I try to
remove that CDN Folder and at the same time i also remove everything
which comes after the extension jpg if has ";" or "?" .
- wlxrs.com
- reverbnation , megavideo , xaxtube
- NOGOMI/ AHLASOOT .. for example to save bandwidth !!! . usually
before you download the MP3 file they allow you to listen online
through their web player which has the same file size and same
file_name but if you download the file then you get a different domain
name which made me think just to rewrite the URL which matches and
save the same URL into cache if a client wanted to listen or download
and also alot of websites has the same idea.
- vkontakte , depositfiles , eporner , 4shared , letitbit , sendspace
, filesonic , uploaded , uploading , turbobit, wupload , redtubefiles
, filehippo , oron , rapishare , tube8 , pornhub , xvideos , telstra ,
scribdassets , przeklej , hardsextube , fucktube , imageshack , beeg ,
yahoo videos , youjizz , gcdn
for example , look at this URL :
#http://b2htks6oia9cm5m4vthd6hhulo.gcdn.biz/d/r/*/FileName WIthout
Extension
(*) DYNAMIC CONTENT and just look into the sub domain how its like!
and so many websites but unfotunately EASY-SHARE is using POST
response and I cant cache it.
Lately I was monitoring NOKIA Phones and I added OVI STORE.
Do you have any website that its not cacheable or using CDN or
something because Im really interested to look at :) .
Ghassan
On Mon, Apr 30, 2012 at 2:53 AM, Eliezer Croitoru<eliezer@xxxxxxxxxxxx> wrote:
On 30/04/2012 02:18, Ghassan Gharabli wrote:
Hello Eliezer,
Are you trying to save all video chunks into same parts or capture /
download the whole video object through CURL or whatever! but i dont
think it should work since it will occure an error with the new
Youtube Player.
What I have reached lately is saving same youtube video chunks that
has youtube 360p itag=34 ,480p itag=35 without saving its itag since
i want to save more bandwidth (thats why i only wrote scripts to you
as an example) which means if someone wants to watch 480p then he
would get the cached 360p contents thats why i didnt add the itag but
if he thought of watching 720p and above then another script would
catch it matching ITAG 37 , 22 ... I know that is not the best
solution but at least its working pretty well with no erros at all as
long as the client can always fast forward .
Im using Squid 2.7 Stable9 compiled on windows 64-bit with PERL x64.
Regarding the 302 Redirection .. I have made sure to update the
source file client_side.c to fix the loop 302 Redirection but really I
dont have to worry now about anything so what is your target regarding
Youtube with argument Range and whats the problem till now ?
I have RAID with 5 HDD and the average HTTP Requests per minute :
2732.6 and because I want to save more bandwidth I try to analyze HTTP
Requests so i can always update my perl script to match most wanted
websites targetting Videos , Mp3 etc.
For a second I thought of maybe someone would help to compile an
intelligent external helper script that would capture the whole
byte-range and I know it is really hard to do that since we are
dealing with byte-range.
I only have one question that is always teasing me .. what are the
comnparison between SQUID and BLUE COAT so is it because it is a
hardware perfromance or just it has more tricks to cache everything
and reach a maximum ratio ?
Ghassan
i was messing with store_url_rewrite and url_rewrite quite some time just
for knowledge.
i was researching every concept exists with squid until now.
a while back (year or more) i wrote store_url_rewrite using java and posted
the code somewhere.
the reason i was using java was because it's the fastest and simples from
all other languages i know (ruby perl python).
i was saving bandwidth using nginx because it was simple to setup.
i dont really like the idea of faking my users about the quality and also it
can make a very problematic state that the user can get partial HQ content
what will break the user from watching videos.
i dont really have any problems with the ranges.
i just noticed that there are providers in my country that are not using the
"range" parameter but the "begin" parameter...
i will be very happy to get the client_side.c patch to fix the 302 loop.
the problem with external_helper is that it's not really needed.
if you need something you can use ICAP for that but you still there is a lot
of work to be dont so as far it seems that store_url_rewrite is the best
option.
BLUECOAT has option to relate objects also using ETAG and other objects
parameters so it makes it a very robust caching system.
nginx does the work just fine for youtube videos but it seems like some
headers are messing and should be added.
by the way,
What video\mp3 sites are you caching using your scripts?
Eliezer
On Mon, Apr 30, 2012 at 1:29 AM, Eliezer Croitoru<eliezer@xxxxxxxxxxxx>
wrote:
On 24/04/2012 21:02, Eliezer Croitoru wrote:
as for some people asking me recently about youtube cache i have checked
again and found that youtube changed their video uris and added an
argument called "range" that is managed by the youtube player.
the original url\uri dosnt include range but the youtube player is using
this argument to save bandwidth.
i can implement the cahing with ranges on nginx but i dont know yet the
way that range works.
it can be based on user bandwidth or "fixed" size of chunkes.
if someone up to the mission of analyzing it a bit more to understand it
so the "range" cache will be implemented i will be happy to get some
help with it.
Thanks,
Eliezer
as for now the "minimum_object_size 512 bytes" wont do the trick for 302
redirection on squid2.7 because the 302 response is 963 big size.
so i have used:
minimum_object_size 1024 bytes
just to make sure it will work.
and also this is a youtube videos dedicated server so it's on with this
limit.
Regards,
Eliezer
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer<at> ngtech.co.il
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer<at> ngtech.co.il
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il
