We've had some instances where crawlers have stumbled onto a cgi script which refers to itself and start pounding the server with requests to that cgi. There are so many CGI scripts on this server that I don't want to maintain a huge robots.txt file. Any suggestions on other techniques to keep crawlers away from cgi scripts? Check the browser with BrowserMatch and then do something creative with "deny from env="? --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|