On Wednesday 25 January 2006 12:30, Anze Vidmar wrote: > On Wed, 2006-01-25 at 18:24 +0000, Sean O Sullivan wrote: > > Look into robots.txt > > Thanks! > > I've terminated them successfully ;-) > Bear in mind that only those robots that choose to play nicely will honor your "robots.txt" files. Some robots simply ignore the file and collect whatever links they can follow. Regards, Mike Klinke -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
