James
On Sun, Jan 17, 2010 at 9:31 AM, Kai Schaetzl <maillists@xxxxxxxxxxxxx> wrote:
Add
User-agent: Slurp
Crawl-delay: 86400
to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at
least follows these rules. Something you can't say of Googlebot, for
instance.
Kai
--
Get your web at Conactive Internet Services: http://www.conactive.com
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
--
http://www.astorandblack.com
--
_______________________________________________ CentOS mailing list CentOS@xxxxxxxxxx http://lists.centos.org/mailman/listinfo/centos
