http://www.redhat.com/robots.txt On 7/11/07, Karl Latiss <karl.latiss@xxxxxxxxxxxxx> wrote:
On Wed, 2007-07-11 at 02:52 +0000, debu wrote: > Hi Group, > > I was just trying to find all the tags that can be used inside apache robots.txt, any link, or any addition to the following? > > i have these, > > User-agent: > Disallow: > Crawl-delay: > Request-rate: > Visit-time: > > Thanks > Debajit kataki According to http://www.robotstxt.org/wc/norobots.html only User-agent and Disallow are valid. All other entries are user-agent specific (see for eg http://www.ilovejackdaniels.com/seo/robots-txt-file/3/) -- Karl Latiss <karl.latiss@xxxxxxxxxxxxx> Atvert Systems -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
-- :. Best Wishes :. Waleed Harbi --------------------------------------------- :. Never too old to learn :. Every why has a wherefore :. Grasp all, lose all -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
