On 6/24/05, Richard Lynch <ceo@xxxxxxxxx> wrote: > On Thu, June 23, 2005 3:37 am, JamesBenson said: > > http://www.funender.com/phpBB2/about18577.html > > Call me crazy, but... > > A) > Doesn't robots.txt have to be in public_html? How the hell can the robots > read it if it's in the root folder, as they suggest in that forum? > > B) > Are the Bad Bots really going to honor robots.txt in the first place? I > mean, I know the spammers don't even need to un-obfuscate emails as simple > as %40 and #&64; but are they really so dumb as to honor robots.txt while > they write their crawlers?... That's crazy. > > Though I guess my theory on that holds true here as well: If you catch a > million fish every time you cast your line, are you gonna try better bait? > No. So they don't un-obfuscate even the simplest email mask, and I guess > some of them play nice as far as robots.txt goes, even though they are out > to spam you. Seems silly, but that could be Reality. > > C) > Aren't some of those "bad" bots also going to rule out legitimate > scripting? Am I mistaken that he pre-emptorally denies access to any > (legit) Python script? Hello? I'm not a fan of Python, but that seems a > bit much. :-) > [Just kidding. I'm ambivalent towards Python, really.] > > -- > Like Music? > http://l-i-e.com/artists.htm > robots.txt SHOULD go in the webroot, like this: http://l-i-e.com/robots.txt I don't know why FuNEnD3R said that it should be above the webroot. That is a mistake. Dotan http://lyricslist.com/lyrics/artist_albums/64/the_beatles.php Beatles Song Lyrics -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
