Add User-agent: Slurp Crawl-delay: 86400 to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at least follows these rules. Something you can't say of Googlebot, for instance. Kai -- Get your web at Conactive Internet Services: http://www.conactive.com _______________________________________________ CentOS mailing list CentOS@xxxxxxxxxx http://lists.centos.org/mailman/listinfo/centos
Re: httpd and robots.txt
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
- To: centos@xxxxxxxxxx
- Subject: Re: httpd and robots.txt
- From: Kai Schaetzl <maillists@xxxxxxxxxxxxx>
- Date: Sun, 17 Jan 2010 15:31:17 +0100
- Delivered-to: centos@xxxxxxxxxx
- In-reply-to: <1E79B0DC7C40472F9D3E741D8D05485B@local>
- References: <1E79B0DC7C40472F9D3E741D8D05485B@local>
- Follow-Ups:
- Re: httpd and robots.txt
- From: James Matthews
- Re: httpd and robots.txt
- References:
- httpd and robots.txt
- From: R-Elists
- httpd and robots.txt
- Prev by Date: Re: Get Me Outta Here! Web site security issue
- Next by Date: Re: OT: free DNS service?
- Previous by thread: Re: httpd and robots.txt
- Next by thread: Re: httpd and robots.txt
- Index(es):
 |