Re: Blatantly Evil Question

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



robots.txt will not do what you want it to.

Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish their UA strings on their websites, AFAIK) and send different content if it's one of those.

they will hammer you for it eventually - AFAICT all major SEs send out their
spiders occasionally with faked user-agent strings - to catch out crap like
this.

google adsense won't. I explicity asked them about this. Well, what I asked was that if I had a password protected area, could I allow them access to spider the content so that normal users could see the ads. I told them the layout would be different, but the content the same.

They said that was fine.

2cents.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php


[Index of Archives]     [PHP Home]     [Apache Users]     [PHP on Windows]     [Kernel Newbies]     [PHP Install]     [PHP Classes]     [Pear]     [Postgresql]     [Postgresql PHP]     [PHP on Windows]     [PHP Database Programming]     [PHP SOAP]

  Powered by Linux