On Sat, Jan 16, 2010 at 10:18 PM, R-Elists <lists07@xxxxxxxxxxxx> wrote:
>
> would anyone out there care to share their robots.txt experience using
> centos as a webserver and their robots.txt files?
>
> i realize this is a somewhat simple exercise, yet i am sure there are both
> large and small hosters out there and possibly those that have high traffic
> modify their robots.txt files differently that others ???
>
> please share if you can or care to please?
>
> for years we have just did a * (allow all) and disallow on things like
> /cgi-bin
>
> as examples of places to visit for those out or in the know...
>
> http://www.robotstxt.org/
>
> http://en.wikipedia.org/wiki/Robots_exclusion_standard
>
> http://www.google.com/robots.txt
>
> and others...
>
> quite frankly, there are many orgs out there that dont follow this anyways,
> right?
Right
http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must-die.html
>
> anyone?
>
> tia
>
> - rh
>
> _______________________________________________
> CentOS mailing list
> CentOS@xxxxxxxxxx
> http://lists.centos.org/mailman/listinfo/centos
>
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
