On Sat, 2010-01-16 at 14:18 -0800, R-Elists wrote: > quite frankly, there are many orgs out there that dont follow this > anyways,right? Since robots.txt is a "suggestion" and .htaccess is actually enforced, I use a simple robots.txt like this: User-agent: * Disallow: and put the bad guys into .htaccess. -- MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com _______________________________________________ CentOS mailing list CentOS@xxxxxxxxxx http://lists.centos.org/mailman/listinfo/centos
Re: httpd and robots.txt
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
- To: CentOS mailing list <centos@xxxxxxxxxx>
- Subject: Re: httpd and robots.txt
- From: Frank Cox <theatre@xxxxxxxxxxx>
- Date: Sat, 16 Jan 2010 16:33:08 -0600
- Delivered-to: centos@xxxxxxxxxx
- In-reply-to: <1E79B0DC7C40472F9D3E741D8D05485B@local>
- Organization: Melville Theatre
- References: <1E79B0DC7C40472F9D3E741D8D05485B@local>
- References:
- httpd and robots.txt
- From: R-Elists
- httpd and robots.txt
- Prev by Date: Re: httpd and robots.txt
- Next by Date: Newer Rsyslog than in distro
- Previous by thread: Re: httpd and robots.txt
- Next by thread: Re: httpd and robots.txt
- Index(es):
 |