Search squid archive

Re: SQUID store_url_rewrite

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 31/05/11 11:54, Ghassan Gharabli wrote:
Hello again,

         #generic http://variable.domain.com/path/filename."ex";, "ext" or "exte"
         #http://cdn1-28.projectplaylist.com
         #http://s1sdlod041.bcst.cdn.s1s.yimg.com
#} elsif (m/^http:\/\/(.*?)(\.[^\.\-]*?\..*?)\/([^\?\&\=]*)\.([\w\d]{2,4})\??.*$/)
{
#        @y = ($1,$2,$3,$4);
#        $y[0] =~
s/([a-z][0-9][a-z]dlod[\d]{3})|((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
#        print $x . "storeurl://" . $y[0] . $y[1] . "/" . $y[2] . "."
. $y[3] . "\n";


Why we had to use arrays in this example.
I understood that m/ indicates a regex match operation , "\n" to break
the line and we assined @y as an array which has
4 values we used to call each one for example we call $1 the first
record as y[0] ..till now its fine for me
and we assign a value to y[0] =~ $y[0] =~
s/([a-z][0-9][a-z]dlod[\d]{3})|((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
...

Please correct me if im wrong here.Im still confused about those
values $1 , $2 , $3 ..
how does the program know where to locate $1 or $2 as there is no
values or $strings anyway
as I have noticed that $1 means an element for example
http://cdn1-28.projectplaylist.com can be grouped as elements .. Hope
Im correct on this one
http://(cdn1-28) . (projectplaylist) . (com) should be http:// $1 . $2 . $3


m//  produces $1, $2, ... $9  for each () element in the pattern.

s// will produce different $1, $2, ... etc. You have to save the ones from m// somewhere if you want to use them after s//. The person who wrote that saves them in the array y[].


Then let me see if I can solve this one to match this URL
http://down2.nogomi.com.xn55571528exgem0o65xymsgtmjiy75924mjqqybp.nogomi.com/M15/Alaa_Zalzaly/Atrak/Nogomi.com_Alaa_Zalzaly-3ali_Tar.mp3

so I should work around the FQDN and leave the rest as is, please if
you found any wrong this then correct it for me
#does that match
http://down2.nogomi.com.xn55571528exgem0o65xymsgtmjiy75924mjqqybp.nogomi.com/M15/Alaa_Zalzaly/Atrak/Nogomi.com_Alaa_Zalzaly-3ali_Tar.mp3
   ??
elsif (m/^http:\/\/(.*?)(\.[^\.\-]*?\..*?)\/([^\?\&\=]*)\.([\w\d]{2,4})\??.*$/)
{
       @y = ($1,$2,$3,$4);
       $y[0] =~ s/[a-z0-9A-Z\.\-]+/cdn/
       print $x . "storeurl://" . $y[0] . $y[1] . "/" . $y[2] . "." .
$y[3] . "\n";


does this example matches Nogomi.com domain correctly ?

and why u used s/[a-z0-9A-Z\.\-]+/cdn/

I only understood that you are mnaking sure to find small letters ,
cap letters , numbers but I believe \. is to search
for one dot only .. how about if there is 2 dots or more that 3 dots
in this case! .. another one u r finding dash ..

That pattern ends with "+". To search for "one or more" of the listed safe domain letters.

It matches all of the $y[0] content:
"down2.nogomi.com.xn55571528exgem0o65xymsgtmjiy75924mjqqybp".

While also not-matching bad things like
  "http://evil.com?url=http://nogomi...";

(the one I gave will covert "http://evil.com?url=http://nogomi..."; -->
"cdn://evil.com?url=http://nogomi..."; )



The only thing im confused about is why we have added /cdn/ since the
url doesnt has a word "cdn"?

This is a "s//" operation. ('s' meaning 'switch'). *IF* the $y[0] value matches the pattern for a domain s// will place "cdn" instead of that matched piece.

So what this does is change *.nogomi.com -->  "cdn.nogomi.com"

If there are any bad stuff like my evil.com example going on it will screw with those URL as well. BUT the bits there will not map to "cdn.nagomi.com" so will not corrupt the actual CDN content.

Thinking about it a bit more I should have been more careful and told you:
  s/^[a-z0-9A-Z\.\-]+$/cdn/

which will ONLY match if the $y[0] as a whole is a valid host name text.


Why we have used storeurl:// because I can see some of examples are
print $x . "http://"; . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

can you give me an example to add the portion of $y[1] please..

elsif (m/^http:\/\/(.*?)(\.[^\.\-]*?\..*?)\/([^\?\&\=]*)\.([\w\d]{2,4})\??.*$/)
 {
   @y = ($1,$2,$3,$4);

   if (m/$y[1]/nagomi.com/) {
     $y[0] =~ s/[a-z0-9A-Z\.\-]+/cdn/
   } else {

$y[0] =~ s/([a-z][0-9][a-z]dlod[\d]{3})|((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;

   }

print $x . "storeurl://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

 }


Which one have your interests , writing a script to match the most
similar examples in one rule or writing each script for each FQDN?

The example you started with had some complex details built into its s// matching. So that particular CDN syntax would be detected and replaced. This is useful if the CDN is only some sub-domains of the main site. And there are other non-CDN subdomains to be avoided. The nasty CDN.


The one I've just put above is for use when the site just uses all its subdomains as CDN for the same content. These are the semi-friendly CDN. You can extend that for other CDN by adding their base domains to the m// test. ie if (m/(nagomi|example)\.com|example\.net/) ...

This is only safe when the subdomain portion has meaning to the CDN operator as:
 a) their client account token
 b) their data center routing tagging
 c) their load-balanced server hostname
 .. or similar internal *routing* details.

If there is any content clash on the URL-path portion it cannot be done like this. As I said at the start you MUST BE CERTAIN of the meaning of the bits you are removing.


The horribly nasty ones (looking at akamai here) need the other end of the CDN domain stripped off. exmaple.com.akamai.com --> example.com

None of these patterns we have talked about so far is suitable for those ones.


for example sometimes we see
http://down2.xn55571528exgem0o65xymsgtmjiy75924mjqqybp.example.com/folder/filename.ext
or
http://cdn.xn55571528exgem0o65xymsgtmjiy75924mjqqybp.example2.com/xn55571528exgem0o65xymsgtmjiy75924mjqqybp/folder/filename.ext

really that is interesting to me , that is why I would love to match
this too as well but the thing is if I knew all of these things ..
everything would be fine for me

That is getting extra complex.

I think you have so far been obfuscating the actual URLs and details for examples. Domain bits are relatively easy due to their limited character set.

Path bits must be coded for particular instances often with exacting knowledge. hello.txt and hello.Txt are not necessarily the same file for example. Bes sure what you "know" is correct, and that your patterns work. Then test, re-test, and test again. Then when its working. re-test.


Again I want to thank you for answering my questions as I felt like Im
writing a magazine heheheh

And you get a book back :)
Welcome. Though this is about as far as I can go on examples and generics. My own skill with regex is not that great. (10 years in and I'm still learning "common knowledge" details about it.)

Amos
--
Please be using
  Current Stable Squid 2.7.STABLE9 or 3.1.12
  Beta testers wanted for 3.2.0.7 and 3.1.12.1


[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux