A misguided person is using the robots.txt exclusion file to search for vulnerable web applications. What he plans on doing with this list of vulnerable web applications is up to debate. What he is doing is a violation of the RFC's (governing robots.txt.. Yes, hackers do that also) The robots.txt file is NOT AN ACCESS CONTROL LIST, and SHOULD NOT BE USED TO 'HIDE' DIRECTORIES. AALL DIRECTORIES SHOULD BE PROTECTED AGAINST Directory listing. The only files and directories in the robots.txt file should be files and directories that are already exposed on the web, and not include any hidden or private directories. Take this opportunity to review your web logs, and robots.txt file to see what normally unpublished information and hints it may expose. If you see WebVulnCrawl.blogspot.com in your logs, you may wish to lodge complaints (after tightening up your security) Further, dshield shows them portscanning the net also, looking for unpublished information on unpublished servers. Spammers harvesting email addresses? (illegal under federal can-spam law) Hackers looking for credit card info? Hackers looking for vulnerable web servers? (WebVuln Crawl?) Either case, illegal under FEDERAL 1990 computer abuse and fraud act, 'attempted access beyond authorization' Several other people also think this is illegal: http://webvulncrawl.blogspot.com/2005/12/what-am-i-doing.html 216.179.125.69 - - [11/Mar/2006:06:03:15 -0500] "GET /js/ HTTP/1.1" 403 213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:18 -0500] "GET /includes/ HTTP/1.1" 403 219 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:20 -0500] "GET /mailman/ HTTP/1.1" 404 214 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:22 -0500] "GET /cgi-bin/ HTTP/1.1" 403 218 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:25 -0500] "GET /icons/ HTTP/1.1" 403 216 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:27 -0500] "GET /ClientList/ HTTP/1.1" 404 217 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:29 -0500] "GET /OtherMedia/ HTTP/1.1" 404 217 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:34 -0500] "GET /testlist.html HTTP/1.1" 404 219 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:36 -0500] "GET /nessus/msiis_testlist.html HTTP/1.1" 200 31073 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:41 -0500] "GET /pipermail HTTP/1.1" 404 215 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [11/Mar/2006:06:03:43 -0500] "GET /errors/ HTTP/1.1" 404 213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:31 -0500] "GET /js/ HTTP/1.1" 403 213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:34 -0500] "GET /includes/ HTTP/1.1" 403 219 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:37 -0500] "GET /mailman/ HTTP/1.1" 404 214 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:39 -0500] "GET /cgi-bin/ HTTP/1.1" 403 218 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:42 -0500] "GET /icons/ HTTP/1.1" 403 216 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:45 -0500] "GET /ClientList/ HTTP/1.1" 404 217 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:47 -0500] "GET /OtherMedia/ HTTP/1.1" 404 217 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:52 -0500] "GET /testlist.html HTTP/1.1" 404 219 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:54 -0500] "GET /nessus/msiis_testlist.html HTTP/1.1" 200 31073 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:48:58 -0500] "GET /pipermail HTTP/1.1" 404 215 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - - [14/Mar/2006:23:49:01 -0500] "GET /errors/ HTTP/1.1" 404 213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" -- Michael Scheidell, CTO 561-999-5000, ext 1131 SECNAP Network Security Corporation Keep up to date with latest information on IT security: Real time security alerts: http://www.secnap.com/news
