On 27/11/2011 5:32 a.m., Ghassan Gharabli wrote:
Hello Amos,
Finally, I have almost captured the most YouTube Videos except
something I want to get some asistance from you .
As I have tested before and tried so many times .. Chudy's script is outdated.
After testinig and logging Youtube Videos . I finally have found
something not being fully cached . If you still remember I have said
before with my old messages that ID isnt being captured in all places
but its okay I have done this . I will post my details after I
completelly finish them.
Could you please explain to me whats happening here?
If&range=13-2375679 was found in a URL then Squid doesnt understand
how to cache the full video .. as it only cache the first 13 seconds I
guess! and then it stops . If I try to download this finished cached
movie then you notice its size about 2.2 MB . You try to remove it
from cache then Squid cant even find it as it claims not cached but
shows TCP_HIT in access.log . STRANGE!
(NP: by remove you mean PURGE request? HUT just means cached data was
found to service the request, which is right since purging the data
involves locating it (HITing) before erasing the cached entry. Followup
requests after the purge should not be HIT.).
I took a look at these"range" replies being generated by YT a while back.
What I found was that a request for video URL would send back a FLV
object with bytes eg "[SWF...]ABCDEFGH". All fine and good this is the
cacheable video.
If the user skips around in the video the player generates a range=
request stating what timestamp or bytes they want to strat at. Its not
clear which due to the reply which comes back having a *different* byte
sequence than the video at the same URL. For example, on the
"[SWF...]ABCDEFGH" video it would produce: "[SWF...]EFGH" or something
similar.
Under the HTTP rules the range object to be combined must be a snippet
portion of the base object (range 4-999, should have been just "DEFGH").
By adding the SWF headers on each reply YT are making them unique and
different objects. Combining them in the middle (ie by a caching app)
will cause errors in the binary object and crash the Flash player or
cause it to display an error message instead of the video
This range request only seems to happen if the user skips into a portion
of video the player has not yet downloaded. So sending them the whole
video, which is what we try to do with Squid, will cause a display lag
for the user but not cause problems in their player.
Now look into this URL:
-------------------------------
"http://o-o.preferred.orange-par1.v4.lscache7.c.youtube.com/videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Csource%2Calgorithm%2Cburst%2Cfactor%2Ccp&fexp=907605%2C912600%2C915002&algorithm=throttle-factor&itag=34&ip=84.0.0.0&burst=40&sver=3&signature=8223490C23E48CB708E04666E4
A550422757CEC6.9D8D78E66DD14FEFC4B5F960F493ED4CDFD7C51C&source=youtube&expire=13
22348400&key=yt1&ipbits=8&factor=1.25&cp=U0hRR1NPVl9FSkNOMV9LSVpFOkpsV3BkS1B1ZXN
F&id=e120643085f56831&range=13-2375679"
HTTP/1.0 200 OK
Last-Modified: Fri, 27 Nov 2009 12:44:54 GMT
Content-Type: video/x-flv
Date: Sat, 26 Nov 2011 16:06:29 GMT
Expires: Sat, 26 Nov 2011 16:06:29 GMT
Cache-Control: private, max-age=24511
Accept-Ranges: bytes
Content-Length: 2375667
X-Content-Type-Options: nosniff
Server: gvs 1.0
X-Cache: MISS from Peer6
X-Cache-Lookup: MISS from Peer6:3128
Connection: close
Whats the job of "Accept_ranges: bytes" here?
Accept-* means the software producing that reply or request supports a
certain HTTP feature. In this case it is Squid and maybe the server as
well supporting HTTP range requests. Not related to YT particulary.
And the very confusion again you can see another similar URL with the
same "/videoplayback?.*(id)" and here comes the ID inthe end of this
URL then moves temporary just . I must mention that this URL sends the
FLV url as Squid already read it in access.log and then it dds
&ir=1&playretry=1 or pr=1&playretry which means Squid would be
confused to cache it 2 times (FLV).
EXAMPLE:
---------------
"http://o-o.preferred.orange-par1.v3.lscache3.c.youtube.com/videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Csource%2Calgorithm%2Cburst%2Cfactor%2Ccp&fexp=908525%2C910207%2C916201&algorithm=throttle
-factor&itag=34&ip=84.0.0.0&burst=40&sver=3&signature=0489805DCC95F6EADBA9D43C3F
D8C107FC768662.73AA6897FE78CF78BE7819E089F1A4FC47534C7D&source=youtube&expire=13
22344800&key=yt1&ipbits=8&factor=1.25&cp=U0hRR1NPUl9FSkNOMV9LSVZJOmdmQWdwWC01dlp
n&id=283246f338ece5ad"
HTTP/1.0 302 Moved Temporarily
Last-Modified: Wed, 02 May 2007 10:26:10 GMT
Date: Sat, 26 Nov 2011 15:50:47 GMT
Expires: Sat, 26 Nov 2011 15:50:47 GMT
Cache-Control: private, max-age=900
Location: http://r9.orange-par2.c.youtube.com/videoplayback?sparams=id%2Cexpire%
2Cip%2Cipbits%2Citag%2Csource%2Calgorithm%2Cburst%2Cfactor%2Ccp&fexp=908525%2C91
0207%2C916201&algorithm=throttle-factor&itag=34&ip=84.0.0.0&burst=40&sver=3&sign
ature=0489805DCC95F6EADBA9D43C3FD8C107FC768662.73AA6897FE78CF78BE7819E089F1A4FC4
7534C7D&source=youtube&expire=1322344800&key=yt1&ipbits=8&factor=1.25&cp=U0hRR1N
PUl9FSkNOMV9LSVZJOmdmQWdwWC01dlpn&id=283246f338ece5ad&ir=1
X-Content-Type-Options: nosniff
Content-Type: text/html
Server: gvs 1.0
Age: 2068
Content-Length: 0
X-Cache: HIT from Peer6
X-Cache-Lookup: HIT from Peer6:3128
Connection: close
This is the 302 redirect Adrian and Chudy were discussing at the end of
the wiki page. If you cache it with storeurl_access reductions it will
loop infinitely back at itself.
Amos
